Migrating from Apache Storm to Apache Flink
Migrating from Apache Storm to Apache Flink
The first thing you need to do is to not just pick up and dump to a new system, but to see what can be reconfigured, refactored or reimagined. For some routing, transformation or simple ingest type applications or solution parts you may want to use Apache NiFi.
For others Spark or Spark Streaming can quickly meet your needs. For simple Thing to Kafka or Kafka to Thing flows, a flow with Kafka Connect is appropriate. For things that need to run in individual devices, containers, pods you may want to move a small application to NiFi Stateless. There are also sometimes a simple Kafka Stream application will meet your needs.
For many use cases you can replace a compiled application with some solid Flink SQL code. For some discussions, check this out.
For some really good information on how to migrate Storm solutions to Flink, Cloudera has a well documented solution for you:
Conceptual
https://docs.cloudera.com/csa/1.2.0/stormflink-migration/topics/csa-stormflink-concept.html
Architecture
https://docs.cloudera.com/csa/1.2.0/stormflink-migration/topics/csa-stormflink-architecture.html
Redistribution
https://docs.cloudera.com/csa/1.2.0/stormflink-migration/topics/csa-stormflink-redistribution.html
References
- https://docs.cloudera.com/csa/1.2.0/development/topics/csa-application-logic.html
- https://www.datainmotion.dev/2020/12/smart-stocks-with-flank-nifi-kafka.html
- https://www.datainmotion.dev/2019/11/introducing-mm-flank-apache-flink-stack.html
- https://www.datainmotion.dev/2020/07/flank-in-cloud-huge-cloudera-data.html
- https://www.datainmotion.dev/2020/05/flank-low-code-streaming-populating.html
- https://www.datainmotion.dev/2020/10/running-flink-sql-against-kafka-using.html
- https://github.com/tspannhw/EverythingApacheNiFi
- https://github.com/tspannhw/ApacheConAtHome2020
- https://github.com/tspannhw/SmartStocks
- https://github.com/tspannhw/ClouderaFlinkSQLForPartners
- https://github.com/tspannhw/SmartWeather
