Migrating Apache Flume Flows to Apache NiFi: Kafka Source to HTTP REST Sink and HTTP REST Source to Kafka Sink

Migrating Apache Flume Flows to Apache NiFi:  Kafka Source to HTTP REST Sink and HTTP REST Source to Kafka Sink

Article 6 - This.
Article 5 - https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_35.html
Article 4 - https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_8.html

Article 3 - https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_7.html

Article 2 - https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html

Article 1 - https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html

Source Code:  https://github.com/tspannhw/flume-to-nifi

This is a simple use case of being a gateway between REST API and Kafka.   We can do a lot more than that in NiFi.  We can be a Kafka Consumer and Producer as well as POST REST calls and receive any REST calls on configurable ports.  All with No Code.

NiFi can act as a listener for HTTP Requests and provide HTTP Responses in a scriptable full Web Server mechanism with JETTY.   Or it can listen for HTTP REST calls on a port and route those files anywhere. https://community.cloudera.com/t5/Community-Articles/Parsing-Web-Pages-for-Images-with-Apache-NiFi/ta-p/248415 .  We can also do websockets https://community.cloudera.com/t5/Community-Articles/An-Example-WebSocket-Application-in-Apache-NiFi-1-1/ta-p/248598.  https://community.cloudera.com/t5/Community-Articles/Accessing-Feeds-from-EtherDelta-on-Trades-Funds-Buys-and/ta-p/248316

It is extremely easy to do this in NiFi.

Kafka Consumer to REST POST

HTTP REST to Kafka Producer

Full Monitoring on Apache NiFi

A Very Common Use Case:  Ingesting Stock Feeds From REST to Kafka

https://community.cloudera.com/t5/Community-Articles/Real-Time-Stock-Processing-With-Apache-NiFi-and-Apache-Kafka/ta-p/249221

References

• https://www.datainmotion.dev/2019/08/creating-apache-kafka-topics.html
• https://community.cloudera.com/t5/Community-Articles/Real-Time-Stock-Processing-With-Apache-NiFi-and-Apache-Kafka/ta-p/249221
• https://community.cloudera.com/t5/Community-Articles/IoT-Edge-Use-Cases-with-Apache-Kafka-and-Apache-NiFi-MiniFi/ta-p/249232
• https://dataworkssummit.com/nosql-day-2019/session/tracking-crime-as-it-occurs-with-apache-phoenix-apache-hbase-and-apache-nifi/
• https://community.cloudera.com/t5/Community-Articles/NiFi-REST-API-FlowFile-Count-Monitoring/ta-p/244208
• https://dzone.com/articles/simple-apache-nifi-operations-dashboard-part-2-spr
• https://dzone.com/articles/building-a-custom-apache-nifi-operations-dashboard
• https://www.progress.com/tutorials/jdbc/connect-and-import-data-from-any-rest-api-in-apache-nifi