The burden of a successful feature: Scaling our real time logging platform
0 likes254 views
The document outlines the challenges and advancements in Fastly's real-time logging platform, which aims to deliver logs rapidly to a variety of endpoints without storing them. It discusses the importance of observability, logging growth metrics, and the impact of system design on performance and guarantees. Future enhancements include improved platform performance, new integrations, and easier logging management for users.
The burden of a successful feature: Scaling our real time logging platform
- 1. Ines Sombra Director of Engineering The burden of a successful feature: Scaling our real time logging platform presents
- 2. Today’s Agenda A delightful demo & context A deep dive into logging Challenges & future
- 3. But first… A bit of context
- 5. https://vimeo.com/267641392 Fresh from Altitude NYC —Peter Bourgon, Altitude NYC Observability is an umbrella term. There are different techniques to achieve observability in a system.
- 6. Peter’s classification of Observability TECHNIQUES SYSTEMS * Lovingly stolen from Peter Bourgon
- 7. SYSTEMS * Lovingly stolen from Peter Bourgon TODAY 👉 Peter’s classification of Observability TECHNIQUES
- 12. But Why? This pipeline is one of the oldest systems at Fastly Born out of our dissatisfaction w the status quo We wanted something that would send you logs extremely fast (stream them near realtime) to anywhere you want (many endpoints)
- 13. Log Streaming at Fastly
- 14. Logging @ Fastly Caches Aggregators Endpoints s3 syslog gcs sumologic bigquery ftp papertrail …
- 15. s3 syslog gcs sumologic bigquery ftp papertrail … Logging @ Fastly Caches Aggregators Endpoints
- 16. s3 syslog gcs sumologic bigquery ftp papertrail … Logging @ Fastly Caches Aggregators Endpoints
- 17. s3 syslog gcs sumologic bigquery ftp papertrail … Logging @ Fastly Caches Aggregators Endpoints
- 18. s3 syslog gcs sumologic bigquery ftp papertrail … Logging @ Fastly Caches Aggregators Endpoints
- 19. s3 syslog gcs sumologic bigquery ftp papertrail … Logging @ Fastly Caches Aggregators Endpoints
- 20. s3 syslog gcs sumologic bigquery ftp papertrail … Logging @ Fastly Caches Aggregators Endpoints
- 21. s3 syslog gcs sumologic bigquery ftp papertrail … Logging @ Fastly Caches Aggregators Endpoints
- 22. Logging pipeline is Stateless We don’t batch your logs We don’t store your logs We stream your logs in near real-time to your defined endpoints We really don’t want your logs on disk
- 23. Logging @ Fastly Caches + Senders Aggregators Varnish Varnish Varnish Varnish
- 24. Varnish Varnish Varnish Varnish Logging @ Fastly Caches + Senders Aggregators
- 25. Varnish Varnish Varnish Varnish Logging @ Fastly Caches + Senders Aggregators
- 26. Varnish Varnish Varnish Logging @ Fastly Caches + Senders Aggregators Varnish
- 27. Logging pipeline is Best Effort We try our best to send logs to your defined endpoint Your endpoint must be up & healthy in order for us to be able to send data to it We have minimal buffering Pipeline optimized for log streaming speed
- 28. Logging Endpoints We don’t limit the number of endpoints or log lines per request ~8.6K active endpoints Ecosystem of endpoints in different stages of evolution Aggregators Endpoints s3 syslog gcs sumologic bigquery ftp papertrail …
- 29. Logging Streams data File-based endpoints (time ranged) Streaming endpoints (protocol or http-requests) s3 gcs ftp sftp syslog sumologic bigquery logentries papertrail splunk scalyr honeycomb
- 30. Logging Growth (2014-2015) 😳 ~430K LPS ~1.2K endpoints ~ 2GBps
- 31. Logging Growth (2014-2015) 😳 ~430K LPS ~1.2K endpoints ~ 2GBps
- 32. Logging Growth (2017-2018) ~3M LPS ~8.6K endpoints ~4GBps
- 33. Logging Growth (2017-2018) ~3M LPS ~8.6K endpoints ~4GBps
- 34. Logging Growth (8X!!) ~3M LPS ~8.6K endpoints ~4GBps
- 36. We send a lot of data continuously to our supported endpoints Syslog continues to be our most popular endpoint but S3 & GCS have the highest volume The 70's are still alive with a very respectable 13 MBps to ftp and 74 kBps to sftp* * for the non-millennials Logging Endpoints
- 37. Challenges & Lessons learned
- 38. s3 syslog gcs sumologic bigquery ftp papertrail … Logging @ Fastly Caches Aggregators Endpoints
- 39. Volume Challenges No hard limits to what you can log, this can be challenging System is multi-tenant. Noisy neighbors can affect delivery Consider sampling for high volume logging
- 40. Burden of many endpoints Classic integrations challenges (each endpoint is a downstream dependency) Standard endpoint clients often don’t meet our needs Having our own clients affords us extra optimizations
- 41. Endpoints & Health Some endpoints have known limitations (infamous examples: S3, BigQuery, GCS) Difficult to infer if an endpoint is working or not (Hard to test setup too) Structured logging (JSON via VCL) is challenging
- 42. Service Isolation Prioritize delivery of content over log retention An aggregator discards the oldest logs it has when it can’t deliver them fast enough In a cache node we are our own customers so senders do the same when they can’t reach aggregators fast enough
- 43. Expectation Mismatch Burden of a system that works so well is that it makes you believe you have strong guarantees Design constraints determine the SLA of the pipeline General advice: Understand the design choices of the systems you use because they limit what is possible to guarantee *
- 45. The team have been Busy bees H2 H1 Platform performance & addressing the challenges of individual endpoints We are getting fancy!
- 46. Platform Performance Reducing lock contention & CPU usage Smarter memory allocation & management Overhauling all endpoints Halving the time it takes for a log line to be processed (from sender read to aggregator line preparation)
- 47. Getting fancy BigQuery improvements New endpoints: Kafka More integrations with cloud services Make endpoints easier to debug
- 49. Want More?
- 50. Want more endpoints? Want metrics? Want easier structured logging? Want VCL counters + secondly aggregation + a higher SLA? Want More?
- 51. Want more endpoints? Want metrics? Want easier structured logging? Want VCL counters + secondly aggregation + a higher SLA? Dom Fee Want More?
- 52. Want more endpoints? Want metrics? Want easier structured logging? Want VCL counters + secondly aggregation + a higher SLA? Dom Fee Want More?
- 53. tl;dr LOGGING Fastly lets you extend the visibility of your system to the edge & gain meaningful insights in near real-time Is a pipeline with very specific constraints & guarantees Exciting things are coming!