BDX 2016- Monal daxini @ Netflix
8 likes•7,558 views
Netflix built a scalable event streaming pipeline called Keystone to replace their legacy system, ingesting over 1 trillion events per day. Keystone utilizes Apache Kafka, Samza, and Spark Streaming to reliably process and route streaming data at massive scale across Netflix's infrastructure. The success of Keystone was due in large part to Netflix's culture of freedom and responsibility that empowered a small team to build and operate the new system without separate management oversight.
BDX 2016- Monal daxini @ Netflix
- 1. The way to deal with Big data problems Monal Daxini March 2016
- 2. Monal Daxini Real Time Data Infrastructure Senior Software Engineer, Netflix https://www.linkedin.com/in/monaldaxini @monaldax #Netflix #Keystone
- 3. We help Produce, Store, Process, Move Events @ scale
- 4. Tell me more... ● Big Data Ecosystem @ Netflix ● How we built a scalable event pipeline - Keystone - in a year ○ Replaced legacy system without service disruption ○ Small team 8 +1 ● Netflix Culture ○ Relevant tenets tagged on the slides
- 5. Global Launch - Jan 6, 2016
- 6. Over 75M Members 190 Countries 125M hours/day → 11B hours / quarter 14,269 years / day → 1,255,707 years / quarter 1000+ devices 37% of Internet traffic at peak
- 7. Netflix Is a Data Driven Company Content Product Marketing Finance Business Development Talent Infrastructure ←CultureofAnalytics→
- 8. Data @ Netflix Data at Rest (batch) Data in Motion (streaming)
- 9. Big Data Systems - batch Ingestion / Kafka -> Ursula, Aegisthus Storage / S3, Teradata, Redshift, Druid Processing / Pig, Hive, Presto, Spark Reporting / Microstrategy, Tableau, Sting Scheduling / UC4 Interface / Big Data Portal, Kragle Opensource& CommunityDriven
- 10. Big Data Systems - batch
- 11. Scale - batch AWS S3 (instead of HDFS) 40 PB (S3) Compressed Of which 13 PB events data
- 12. Big Data Systems - streaming Data Pipeline - Keystone Playback & operational insight - Mantis Stream Processing* - Spark Streaming Metrics & monitoring - Atlas LooselyCoupled HighlyAligned Opensource& CommunityDriven
- 13. What does culture have to do with big data?
- 14. Netflix Culture Deck Netflix Culture Freedom & Responsibility
- 15. "It may well be the most important document ever to come out of the Valley." 1 Sheryl Sandberg COO, Facebook 1 Business Insider, 2013
- 16. A NETFLIX ORIGINAL SERVICE How we built an internal facing 1 trillion / day stream processing cloud platform in a year, and how culture played a pivotal role Freedom & Responsibility
- 17. Years ago...
- 18. In the Old Days ... EMR Event Producers
- 19. Chukwa/Suro + Real-Time Branch
- 20. About a year ago ...
- 21. Chukwa / Suro + Real-Time Branch Event Producer Druid Stream Consumers EMR Consumer Kafka Suro Router Event Producer Suro Kafka Suro Proxy
- 22. Support at-least-once processing Scale, Ease of Operations Replace dormant open source software - Chukwa Enable future value adds - Stream Processing As a Service Seamless transition to the new platform ContextNotControl
- 23. Migrate Events to a new Pipeline In flight, while not losing more that 0.1% of them ContextNotControl HighlyAligned LooselyCoupled
- 24. Jan 2016
- 26. 1 trillion events ingested per day during holiday season 1+ trillion events processed every day 350 billion a year ago 600+ billion events ingested per day Keystone - Scale - Streaming
- 27. 11 million events (24 GB per second) peak Upto 10MB payload / Avg 4K 1.3 PB / day Keystone - Scale - Streaming
- 30. Event Payload is Immutable At-least-once semantics* * Once the event makes it to Kafka, ther are disaster scenarios where this breaks.
- 31. Injected Event Metadata ● GUID ● Timestamp ● Host ● App
- 32. Keystone Extensible Wire Protocol ● Backwards and forwards compatibility ● Supports JSON, AVRO on the horizon ● Invisible to source & sinks ● Efficient - 10 bytes overhead per message ○ because message size - hundreds of bytes to 10MB
- 33. Netflix Kafka Producer ● Best effort delivery - ack = 1 ● Prefer drop event than disrupting producer app ● Resume event production after Kafka cluster restore ● Integration with Netflix Ecosystem ● Configurable topic to Kafka clusters route
- 36. ● Pioneer Tax ● Started with 0.7 ● In prod with 0.8.2 ● Move to 0.9 & VPC in progress Kafka in the Cloud
- 37. Based on topics assigned ● Normal-priority (majority) ● High-priority (streaming activities etc.) Fronting Kafka Topic Classification
- 38. ● ≅3200 d2.xl brokers for regular, failover, & consumer ● 125 Zookeeper nodes ○ Independent zookeeper cluster per Kafka cluster ● 24 island clusters, 8 per region ○ 3 ASGs per cluster, 1 ASG per zone ○ 24 warm standby 3 node failover clusters Scale - Kafka (prod)
- 39. ● No dynamic topic creation ● Two copies ● Zone aware assignment of Topic partitions and replica Fronting Kafka Topics
- 40. In a distributed system make sure you understand limitations and failures, even if you don’t know all the features. - Monal
- 41. In addition, we do Kafka Kong once a week
- 42. Fronting Kafka Failover Self Service Tool BlamelessCulture
- 45. Kafka Management UI (Beta) Open sourcing on the road map Opensource& CommunityDriven
- 48. Kafka Auditor Open sourcing on the road map Opensource& CommunityDriven
- 49. Kafka Auditor - One pre cluster ● Broker monitoring ● Consumer monitoring ● Heart-beat & Continuous message latency ● On-demand Broker performance testing ● Built as a service deployable on single or multiple instances
- 50. Kafka Cluster Size -Tips ● Per Cluster Stay under 10k partitions & 200 brokers ● Leave approx. 40% free disk space on each broker
- 51. ● Started with AWS zone aware partition assignments ● We have discovered and filed several bugs ○ Details - Upcoming in Netflix Tech blog Kafka Contributions Opensource& CommunityDriven
- 52. Routing Service
- 55. Router Job Manager (Control Plane) EC2 Instances Zookeeper (Instance Id assignment) Job Job Job ksnode Checkpointing Cluster ASG
- 56. Custom Go Executor ./runJob Logs Snapshots Attach Volumes ./runJob ./runJob Reconcile Loop - 1 min Health Check What’s running in ksnode? Zookeeper (Instance Id assignment)
- 57. Logs ZFS Volume Snapshots Custom Go Executor . /runJo b . /runJo b . /runJo b Go Tools Server Client Tools Stream Logs Browse through rotated logs by date Ksnode Tooling
- 58. Yes! You inferred right! No Mesos & No Yarn
- 59. Distributed Systems are Hard Keep it Simple Minimize Moving Parts
- 60. ● 13,000 docker containers (samza jobs) ○ 7,000 - S3 Sink ○ 4,500 - Consumer Kafka sink ○ 1,500 - Elasticsearch sink ● 1,300 AWS C3-4XL instances Scale - Routing Service
- 61. More Info - Samza Meetup (10/2015) Samza ver 0.9.1 Contributions Opensource& CommunityDriven
- 62. Target & Achieved <= 0.1% diff bw Chukwa & Keystone pipeline, over 2.6 PB of data / day Chukwa & Keystone Pipeline Shadowing
- 65. Customer Facing per topic end-to-end dashboard
- 66. Dev facing infrastructure end-to-end dashboard
- 67. Scaling Avenues
- 68. ● Exposed cost attribution per event producers & topic ○ E.g. one producer reduced throughput by 600% ● Automation - frees up additional resources Scaling Up by Scaling Down
- 69. ● No dedicated product or project managers ● No separate devops or operational team ● This does not mean we are constantly overworked ○ we make wise and simple choices and ○ lean towards automation & self-healing systems. We build and run what you saw today! YoubuildIt! Yourunit! HighPerformance
- 70. Not DevOps, but move towards NoOps You build it! You run it!
- 71. ● High Performance culture ● Communication ● No culture of process adherence ○ Creativity & Self Discipline ○ Freedom and Responsibility
- 72. Looking into the future?
- 73. Streaming Processing As a Service ● multi-tenant polyglot support of streaming engines like Spark Streaming, Mantis, Samza, and may be Flink Future steps Opensource& CommunityDriven
- 74. Messaging As a Service ● Kafka & Others ● Spark Streaming, Mantis, Samza, and may be Flink. Future steps Opensource& CommunityDriven
- 75. Data thruway ● Support for schemas - registry, discovery, validation. Self Service Tooling Future steps Opensource& CommunityDriven
- 76. More brain food... Netflix OSS Samza Meetup Presentation Netflix Tech Blog Spark Summit 2015 Talk