Don’t Be Scared: Multi-Tenant Cluster Support at Scale (Kelly Attaway, Pandora Media and Cabe Waldrop, Pandora Media) Kafka Summit London 2019
0 likes•902 views
The document discusses the challenges faced by Pandora in managing thousands of unsupervised clients on their Kafka clusters, detailing the issues of performance degradation and client behavior. It outlines their multi-tenant Kafka setup, the performance of their primary production cluster, and the importance of tuning both brokers and clients to optimize performance. Additionally, the document emphasizes lessons learned from real-life experiences and adjustments made to configurations for improved efficiency.
Don’t Be Scared: Multi-Tenant Cluster Support at Scale (Kelly Attaway, Pandora Media and Cabe Waldrop, Pandora Media) Kafka Summit London 2019
- 3. We’re going to: - Describe how thousands of unsupervised clients can wreak havoc on your shared Kafka cluster, other clients, your mind, your body, and your soul - Share how we have lived with and grown from this experience (broker, file system, and client tuning) - Encourage you to learn from our mistakes (spoiler: trust no one to make a well-behaved client)
- 5. Kafka at Pandora - 8 clusters at Pandora of varying size, all multi-tenant - 7 on-prem KC clusters - 4 GCP KC clusters - We have clusters secured with Kerberos and Ranger for secure applications as well as more permissive clusters for general use
- 6. Kafka at Pandora Our primary production cluster: - 15 brokers, v 0.9.0.0 through 1.1.0 - 256GB Memory, 10 GB/s NIC, 24 cores - Disk: 26 1TB disks formatted as 2x raidz2 ZFS - 500 topics/1500 consumer groups/?? producers - 200k msgs/sec, .5TB/day - 2.3M requests/sec
- 7. Sorry How Many Clients?
- 8. Kafka at Pandora - Primary data hub - You got a message at point A? We’ll get it to point B - Analytics pipeline (every play, skip, ad interaction, engagement, etc) sinking to HDFS and GCS - Countless other applications
- 9. The Dream of Multi-Tenancy Low barrier to entry! Speed to production! Low response times for requests (especially produce, fetch, heartbeat)! Ease of monitoring clients! Everyone living in peace and harmony!
- 13. The Thing Noun An occurrence on a Kafka cluster wherein performance drastically degrades on one or more brokers, typically characterized by slow response times, full request queues, and weeping cluster admins. “Why are Kelly and Cabe sobbing?” “Oh, the cluster is doing The Thing again.”
- 14. …
- 17. Ah, so it’s a volume issue?
- 18. LOL! Nope!
- 20. High Volume Cluster - No Degradation Our higher volume prod cluster consistently outperforms our lower volume one with more than double the volume A few hundred well-written producers, batching appropriately A couple dozen well-written consumers, with lag of 0
- 24. The Thing - Revised Definition - Fetch requests for data out of ZFS cache - Network threads blocking while reading from disk - Request queues filling - Response times rising - Too many requests!!!
- 25. Request Handling -- Deep Dive
- 28. The Reality of Multi-Tenancy - Rampant URP - Seemingly inexplicable ISR thrashing - Stifled produce and fetch requests for clients - High response times from the brokers - Death, Destruction, Etc
- 32. Well, what did you do about it?
- 33. Make ZFS Work For You ZFS can make Kafka incredibly fast - Clever about invalidating the cache - Smart about pre-fetching sequential reads - One-off reads don’t pollute your cache But you’ve got to know how to use it...
- 34. Make ZFS Work For You Current Configs (good, yes) - atime=off - vdevs=100 - zfs_vdev_async_read_max_active =30 - zfs_vdev_async_write_max_active=100 - zfs_vdev_sync_read_max_active=100 - zfs_vdev_sync_write_max_active=100 - zfs_vdev_max_active=4000 - zfs_arc_max=214748364800 Original Configs (bad, no) - atime=on - vdevs=10 - zfs_vdev_async_read_max_active =3 - zfs_vdev_async_write_max_active=10 - zfs_vdev_sync_read_max_active=10 - zfs_vdev_sync_write_max_active=10 - zfs_vdev_max_active=1000 - zfs_arc_max=0
- 35. Broker Tuning - Network threads == number of cores - IO threads == number of vdevs - Request queue = bigger (but not too big) - Arc cache = as big as you can
- 36. And you lived happily ever after?
- 43. There’s hope! Reform clients, especially consumers Producer and consumer configs generally follow the same pattern - make wrappers! Save yourselves! It’s too late for us, but you have your whole life before you! Up next in our queue: quotas! L2 cache to ameliorate worst case scenarios
- 44. Tales From The Crypt What configs help and what patterns absolutely do not Stories from real life: - Producing one message a time - Max.poll.interval.ms = Long.MAX_VALUE - Committing consumer offsets for every record - Consuming from the beginning at every group rebalance
- 46. Kafka Is Excellent At High Volume. Clients Are Disrespectful!
- 47. ● KBrowse - a UI to deserialize and view messages in your topic ● Algorithm and Blues - the Pandora Engineering blog ● KAFKA-7504 - performance degradation and patch for ext (not zfs)