Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Flink Job Downtime
1 like478 views
The document outlines the design and manual migration steps for multi-cluster Kafka source in Apache Flink, detailing the necessary tasks for swapping and upgrading clusters while minimizing downtime. It highlights the challenges of scaling multiple Kafka clusters across hybrid cloud environments, emphasizing complex operability and failover issues. The document also discusses benefits such as automated migrations and future work opportunities regarding watermark alignment and optimizations.
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Flink Job Downtime
- 1. Mason Chen | Apple Multi Cluster Kafka Source THIS IS NOT A CONTRIBUTION
- 2. Agenda Motivation FLIP 27 Kafka Source Source Design Example
- 5. Manual Migration Steps Bring up new cluster
- 6. Manual Migration Steps Swap producer
- 7. Manual Migration Steps Wait for consumer to drain
- 8. Manual Migration Steps Source uid and cluster change
- 9. Manual Migration Steps Upgrade with non restore state
- 10. Manual Migration Steps Increase parallelism for lag
- 11. Manual Migration Steps Revert to steady state
- 12. Manual Migration Steps When can we remove nonactive cluster?
- 13. User Manual Migration Steps • Change source uid • Change bootstrap server • Upgrade application • With non restore state • Change parallelism and resources to catch with lag • Revert to steady state when caught up
- 14. Manual Migration Steps • Application downtime • Need to increase system resources for catchup • User manual toil • User could have 100+ jobs • Multiple hours of team coordination Drawbacks
- 15. Scaling Multiple Kafka Clusters • Hybrid cloud: on-prem, private cloud and public cloud providers • Scalability • Topic sharding • Operability and Failover • In place upgrade is complex and error prone
- 16. Agenda Motivation FLIP 27 Kafka Source Source Design Example
- 22. FLIP 27 Kafka Source
- 23. FLIP 27 Kafka Source
- 24. FLIP 27 Kafka Source
- 25. FLIP 27 Kafka Source
- 26. Agenda Motivation FLIP 27 Kafka Source Source Design Example
- 27. Kafka Metadata Service • KafkaStream • Logical abstraction to physical clusters and topics • describeStreams(Collection<String> streamIds); • Pluggable implementation • File based configmap
- 28. Multi Cluster Kafka Source Runtime
- 29. Multi Cluster Kafka Source Runtime
- 30. Multi Cluster Kafka Source Runtime
- 31. Multi Cluster Kafka Source Runtime
- 32. Multi Cluster Kafka Source Runtime
- 33. Multi Cluster Kafka Source Runtime
- 34. Multi Cluster Kafka Source Runtime
- 35. Multi Cluster Kafka Source Runtime
- 36. Multi Cluster Kafka Source Runtime
- 37. Multi Cluster Kafka Source Runtime
- 38. Multi Cluster Kafka Source Runtime
- 39. Multi Cluster Kafka Source Runtime
- 40. Multi Cluster Kafka Source Runtime
- 41. Extension of FLIP 27 Major Components • Kafka Source components • Polling, commit, checkpoint, split assignment, • Source Event RPC • Enumerator Context Proxy • Split assignment and wrapping cluster info • Context thread pools
- 42. Agenda Motivation FLIP 27 Kafka Source Source Design Example
- 43. Migration with Multi Cluster Kafka Source
- 44. Migration with Multi Cluster Kafka Source Initial metadata
- 45. Migration with Multi Cluster Kafka Source Bring up new cluster
- 46. Migration with Multi Cluster Kafka Source Bring up new cluster
- 47. Migration with Multi Cluster Kafka Source Add new cluster metadata
- 48. Migration with Multi Cluster Kafka Source Reconcile metadata
- 49. Migration with Multi Cluster Kafka Source Reconcile metadata
- 50. Migration with Multi Cluster Kafka Source Remove old cluster
- 51. Migration with Multi Cluster Kafka Source Reconcile metadata
- 52. Migration with Multi Cluster Kafka Source Reconcile metadata
- 53. Migration with Multi Cluster Kafka Source Remove old cluster
- 54. User Cluster Migration Steps
- 55. Multi Cluster Kafka Source Benefits • Migrations and failover automated transparently within source • Simplify operations between compute and storage infra • Hybrid Source compatible • Can be leveraged for topic migration
- 56. Future Work • Integrate with split level watermark alignment • Optimizations to remove only affected readers • FLIP-246 (https://cwiki.apache.org/confluence/display/FLINK/ FLIP-246%3A+Multi+Cluster+Kafka+Source)
- 57. Q&A