FLANK Stack: NiFi Processor for Kafka Consumption on Demand

January 18, 2020

FLANK Stack: NiFi Processor for Kafka Consumption on Demand - REST Proxy Example

Writing a Custom Kafka Rest Proxy in 4 Hours

A custom processor for using NiFi as a REST Proxy to Kafka is very easy. So I made one in NiFi 1.10. It's a simple Kafka smart client that accepts POSTs, GETs or whatever HTTP request and returns a message from a Kafka topic, topic can be set via variables, HTTP request or your choice. To get on of the query parameters, you do so like this: ${http.query.param.topic}. I am using the plain old KafkaConsumer class.

If you need data, just CURL it or use your HTTP/REST client controls in Java, Python, Go, Scala, Ruby, C#, VB.NET or whatever.

curl http://localhost:9089?topic=bme680

Download a pre-built NAR and install. Note: this is a pre-release Alpha that I quickly built and tested on a few clusters. This is not an official project or product. This is a POC for myself to see how hard it could it be. It's not! Roll your own or join me in building out an open source project for one. What requirements do you have? This worked for me, send a curl, get a message.

Build a topic for Kafka with SMM in seconds

Here's an entire Kafka REST Proxy in a few steps.

HandleHttpRequest (We could have hundreds of options here.
RouteOnAttribute

Provenance For An Example Kafka REST Call

To use the processor, we need to set some variables for Kafka Broker, Topic, Offset Reset, # of Records to grab at a time, Client Id, Group Id - important for keeping your offset, auto commit, deserializer for your key and value types - String is usually good, maybe Byte.