Creating Apache Kafka Topics Dynamically As Part of a DataFlow
Creating Apache Kafka Topics Dynamically As Part of a DataFlow
Sometimes when you are ingesting data at scale, whether it is from a Data Warehouse, Logs, REST API, IoT, Social Media or other sources, you may need to create new Apache Kafka topics depending on the type, variations, newness, schema, schema version or other changes.
Instead of having to manually create an Apache Kafka topic with Cloudera Streams Messaging Manager or Apache Kafka command line kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test), I would like to create it mid-stream based on names that are relevant to arriving data. So this could be the name of the schema from the data, the table name of the origin date, some unique name generated with the data or another source. For my example, I am generating a unique name via Apache NiFi Expression Language:
nifi${now():format('yyyyMMddmmss')}${UUID()}
This is a Proof of Concept, there are more features I would add if I wanted this for production use cases such as adding fields for Number Of Partitions and Number of Replicas.
Example Run
The processor is very easy to use, you merely enter your Kafka Broker URL, such as demo.hortonworks.com:6667. And then the name of your Kafka topic. The processor will validate to make sure you have a valid name which should be Alphanumeric with only the addition of periods, dashes and underscores. It will run quickly and when completed, you can check out the results. Your flowfile will be unchanged, but you will get new attributes as seen below.
You will get kafka.bootstrap (your Broker URL), kafka.client.id (a generate one time use client id), kafka.topic.<TOPIC_NAME> - with one for each Kafka topic that exists, kafka.topic.creation.success - a status of flag, kafka.topic.message - a message, kafka.topic.YourNewNamed one.
In IntelliJ I quickly developed this program using the Apache Kafka Admin API and some JUnit tests.
For a production use case I would probably just use the Cloudera SMM REST API to create topics.
