Elasticsearch in Netflix
80 likes41,218 views
The document discusses Netflix's use of Elasticsearch for querying log events. It describes how Netflix evolved from storing logs in files to using Elasticsearch to enable interactive exploration of billions of log events. It also summarizes some of Netflix's best practices for running Elasticsearch at scale, such as automatic sharding and replication, flexible schemas, and extensive monitoring.
![So We Evolved
hgrep -C 10 -k 5,2,3 'users.*[1-9]{3}' *catalina.out s3//bucket](https://image.slidesharecdn.com/elasticsearchinnetflix-140124100601-phpapp01/85/Elasticsearch-in-Netflix-31-320.jpg)
![So We Evolved
hgrep -C 10 -k 5,2,3 'users.*[1-9]{3}' *catalina.out s3//bucket](https://image.slidesharecdn.com/elasticsearchinnetflix-140124100601-phpapp01/85/Elasticsearch-in-Netflix-32-320.jpg)
![So We Evolved
hgrep -C 10 -k 5,2,3 'users.*[1-9]{3}' *catalina.out s3//bucket](https://image.slidesharecdn.com/elasticsearchinnetflix-140124100601-phpapp01/85/Elasticsearch-in-Netflix-33-320.jpg)
![So We Evolved
hgrep -C 10 -k 5,2,3 'users.*[1-9]{3}' *catalina.out s3//bucket](https://image.slidesharecdn.com/elasticsearchinnetflix-140124100601-phpapp01/85/Elasticsearch-in-Netflix-34-320.jpg)
![So We Evolved
hgrep -C 10 -k 5,2,3 'users.*[1-9]{3}' *catalina.out s3//bucket
select * from log_events where dateint=20140101](https://image.slidesharecdn.com/elasticsearchinnetflix-140124100601-phpapp01/85/Elasticsearch-in-Netflix-35-320.jpg)
Elasticsearch in Netflix
- 1. Elasticsearch In Netflix Danny Yuan, Jae Bae
- 2. Welcome Hashtag: #ES_in_Netflix @Elasticsearch - Elasticsearch ! @stonse - Sudhir Tonse ! @g9yuayon - Danny Yuan ! @metacret - Jae Bae
- 3. Who Are We?
- 4. Who Are We? Software engineers in Netflix’s Platform Engineering team, working on large scale data infrastructure
- 5. Who Are We? Software engineers in Netflix’s Platform Engineering team, working on large scale data infrastructure Building and operating Netflix’s cloud real-time query service
- 6. Why Are We Here?
- 7. Why Are We Here? How We Use Elasticsearch
- 8. Why Are We Here? How We Use Elasticsearch Why Elasticsearch
- 9. Why Are We Here? How We Use Elasticsearch Why Elasticsearch How We Run Elasticsearch
- 10. Why Are We Here? How We Use Elasticsearch Why Elasticsearch How We Run Elasticsearch To Seek Your Feedback
- 11. How We Use Elasticsearch
- 12. Querying Log Events Tracking Service Deployments
- 14. A Little Historical Perspective
- 15. Netflix is a log generating company that also happens to stream movies - Adrian Cockroft photo credit: http://www.flickr.com/photos/decade_null/142235888/sizes/o/in/photostream/
- 22. Things Changed
- 26. 70,000,000,000
- 27. 1,500,000
- 28. Making Sense of Billions of Events
- 29. So We Evolved
- 30. So We Evolved
- 31. So We Evolved hgrep -C 10 -k 5,2,3 'users.*[1-9]{3}' *catalina.out s3//bucket
- 32. So We Evolved hgrep -C 10 -k 5,2,3 'users.*[1-9]{3}' *catalina.out s3//bucket
- 33. So We Evolved hgrep -C 10 -k 5,2,3 'users.*[1-9]{3}' *catalina.out s3//bucket
- 34. So We Evolved hgrep -C 10 -k 5,2,3 'users.*[1-9]{3}' *catalina.out s3//bucket
- 35. So We Evolved hgrep -C 10 -k 5,2,3 'users.*[1-9]{3}' *catalina.out s3//bucket select * from log_events where dateint=20140101
- 37. Log data Server Farm Log data Server Farm Log Collectors Log data Server Farm
- 43. What Could Go Wrong?
- 45. You thought parallelization would save the day? Think again
- 46. You thought parallelization would save the day? Think again
- 47. What Is Missing?
- 49. Functional Requirements Arbitrary Boolean Queries Aggregated Query - Top N Query - Trend - Distribution
- 50. Non-Functional Requirements - Interactive (response within seconds) ! - Quickly locates the right log events - Minimal programming effort
- 51. It’s All about Extracting Small Data Out of Big Data
- 55. Now Back to the Use Case
- 57. Guided Debugging in the Right Context
- 58. Guided Debugging in the Right Context
- 59. Guided Debugging in the Right Context
- 60. Guided Debugging in the Right Context
- 61. Guided Debugging in the Right Context
- 62. Guided Debugging in the Right Context
- 63. Guided Debugging in the Right Context
- 64. Guided Debugging in the Right Context
- 65. A Useful Pattern
- 66. Aggregated Query -> Individual Query
- 67. Examples - S3 diagnostics ! - Tracking email campaigns - Request traces
- 68. Status:200 RequestId Parent Id Node Id Service Name Status 4965-4a74 0 123 Edge Service 200 4965-4a74 123 456 Gateway 200 4965-4a74 456 789 Service A 200 4965-4a74e 456 abc Service B 200
- 70. Edge Service (456) ---> Gateway (789) Data Name Value Request ID 4965-4a74 Response Time 25 ms Endpoints /rest/service Status Code 200
- 72. Automatic Sharding and Replication
- 74. Flexible Schema
- 76. Flexible Schema - Schemaless - Reasonable defaults
- 79. Nice Extension Model - Customizable REST Actions
- 80. Nice Extension Model - Customizable REST Actions - Site Plugins
- 81. Nice Extension Model - Customizable REST Actions - Site Plugins - River Plugins
- 82. Nice Extension Model - Customizable REST Actions - Site Plugins - River Plugins - Discovery Module
- 84. Ecosystem - Plugins, Kibana
- 86. ! { edda }
- 88. Built by Netflix Monitoring Eng Team
- 89. Built by Netflix Monitoring Eng Team Tracks History and Changes to Service Deployments
- 90. Built by Netflix Monitoring Eng Team Tracks History and Changes to Service Deployments Keeps Many Revisions
- 91. Built by Netflix Monitoring Eng Team Tracks History and Changes to Service Deployments Keeps Many Revisions Tracks Dozens of Document Types
- 96. Schemas may change at any time
- 97. Schemas may change at any time Go schemaless
- 99. Users may search for any combination of fields
- 100. Users may search for any combination of fields This is what search engine is designed for
- 102. Users often needs only a few fields
- 103. Users often needs only a few fields Projection via “fields” query
- 105. Need range queries on date and revisions
- 106. Need range queries on date and revisions Natively supported by Elasticsearch
- 107. Need range queries on date and revisions Natively supported by Elasticsearch Route by document ID
- 108. Running ES in Netflix
- 110. Operational Challenges Back pressure when indexing
- 111. Operational Challenges Back pressure when indexing Diverse configurations and data
- 112. Operational Challenges Back pressure when indexing Diverse configurations and data Dynamic flow of log events
- 113. Operational Challenges Back pressure when indexing Diverse configurations and data Dynamic flow of log events Needs extensive monitoring and alerting
- 114. Operational Challenges Back pressure when indexing Diverse configurations and data Dynamic flow of log events Needs extensive monitoring and alerting Tolerating outage at different scales
- 115. Favor Pulling Over Pushing
- 118. Choose Config with Data
- 120. Integrating ES
- 121. AMI for Deployment by Asgard
- 123. Eureka for Server Discovery
- 124. Suro for Data Delivery
- 125. Servo for Monitoring Metrics
- 128. Multi-region Deployment Discovery over Cassandra Region-aware replication
- 129. Favor Index Rolling Over TTL
- 130. Favor Index Rolling Over TTL A dedicated service manages index rolling Uses index template and routing
- 131. Worth Trying G1
- 132. Worth Trying G1 Not recommended by ES team, but
- 133. Worth Trying G1 Not recommended by ES team, but Has fewer and shorter GC pauses
- 134. Worth Trying G1 Not recommended by ES team, but Has fewer and shorter GC pauses Occasional SIGSEGV, but it’s okay
- 135. Simple Majority for Master Election
- 136. Simple Majority for Master Election Split-brain problem
- 137. Simple Majority for Master Election Split-brain problem discovery.zen.minimum_master_nodes
- 138. Simple Majority for Master Election Split-brain problem discovery.zen.minimum_master_nodes Dynamically updated
- 139. Future Work
- 140. Future Work Automatic incremental backup and restore
- 141. Future Work Automatic incremental backup and restore Auto scaling
- 142. Future Work Automatic incremental backup and restore Auto scaling Fully automated deployment
- 143. Future Work Automatic incremental backup and restore Auto scaling Fully automated deployment Support more use cases
- 144. We’re Hiring
- 145. Thank You!