Influx/Days 2017 San Francisco | Christine Yen
0 likes•128 views
The document discusses the importance of observability in the development process, emphasizing the ability to understand a system's internal state through its external outputs and data. It outlines a structured approach that includes deciding, building, verifying, and monitoring to help answer critical questions about system performance and customer experiences. The discussion also highlights the use of flexible observability tools and best practices for gathering contextual and structured data effectively.
Influx/Days 2017 San Francisco | Christine Yen
- 1. OBSERVABILITY: NOT JUST AN OPS THING @cyen @honeycombio
- 2. OBSERVABILITY THE DEV PROCESS
- 3. OBSERVABILITY THE DEV PROCESS The practice of understanding the internal state of a system via knowledge of its external outputs. Wikipedia (paraphrased)
- 4. OBSERVABILITY THE DEV PROCESS Twitter hive mind
- 5. OBSERVABILITY THE DEV PROCESS The ability to answer questions about your system, using data
- 6. OBSERVABILITY THE DEV PROCESS The ability to answer questions about your system, using data DECIDE IT
- 7. OBSERVABILITY THE DEV PROCESS The ability to answer questions about your system, using data DECIDE IT BUILD IT
- 8. OBSERVABILITY THE DEV PROCESS The ability to answer questions about your system, using data DECIDE IT VERIFY IT BUILD IT
- 9. OBSERVABILITY THE DEV PROCESS The ability to answer questions about your system, using data DECIDE IT VERIFY IT (ON MY MACHINE) BUILD IT
- 10. OBSERVABILITY THE DEV PROCESS The ability to answer questions about your system, using data DECIDE IT …? VERIFY IT (ON MY MACHINE) BUILD IT
- 11. OBSERVABILITY THE DEV PROCESS The ability to answer questions about your system, using data DECIDE IT BUILD IT VERIFY IT (ON MY MACHINE) VERIFY IT (IN PRODUCTION)
- 12. OBSERVABILITY THE DEV PROCESS The ability to answer questions about your system, using data DECIDE IT BUILD IT VERIFY IT (ON MY MACHINE) VERIFY IT (IN PRODUCTION) WATCH IT
- 13. OBSERVABILITY THE DEV PROCESS The ability to answer questions about your system, using data DECIDE IT BUILD IT VERIFY IT (ON MY MACHINE) VERIFY IT (IN PRODUCTION) WATCH IT
- 14. OBSERVABILITY THE DEV PROCESS The ability to answer questions about your system, using data DECIDE IT BUILD IT VERIFY IT (ON MY MACHINE) VERIFY IT (IN PRODUCTION) WATCH IT
- 15. OBSERVABILITY THE DEV PROCESS How do my error rates look these days? Have things gotten slower over time?
- 16. OBSERVABILITY THE DEV PROCESS How do my error rates look these days? Have things gotten slower over time? What does my service look like from this one annoying customer’s perspective?
- 17. OBSERVABILITY THE DEV PROCESS How do my error rates look these days? Have things gotten slower over time? What will the impact be, of this change we’re planning? What does my service look like from that one huge, annoying customer’s perspective?
- 18. OBSERVABILITY THE DEV PROCESS How do my error rates look these days? Have things gotten slower over time? What will the impact be, of this change we’re planning? What does "normal" look like these days? Does it line up with what I thought? What does my service look like from that one huge, annoying customer’s perspective?
- 19. OBSERVABILITY THE DEV PROCESS The ability to answer questions about your system, using data DECIDE IT BUILD IT VERIFY IT (ON MY MACHINE) VERIFY IT (IN PRODUCTION) WATCH IT &
- 20. WHY DOES THIS MATTER SO MUCH TO ME?
- 21. ▸ How’s our load? Is it spread reasonably evenly across our Kafka partitions? ▸ Did latency increase in our API server? How does our new batching endpoint compare to our old RESTy endpoint? ▸ How did those recent memory optimizations affect our query- serving capacity?
- 22. ▸ How’s our load? Are high-volume customers spread reasonably evenly across our Kafka partitions? ▸ Did latency increase in our API server? Which customers were impacted the most? And who’ll benefit the most from batching? ▸ How did those recent memory optimizations affect our query- serving capacity for customers with string-heavy payloads?
- 24. OK. SO WHAT DOES THIS LOOK LIKE?
- 25. DECIDE IT BUILD IT VERIFY IT (WFM🤘) VERIFY IT (IN PROD) WATCH IT
- 26. BUILD IT DECIDE IT VERIFY IT (WFM🤘) VERIFY IT (IN PROD) WATCH IT
- 27. BUILD IT DECIDE IT VERIFY IT (WFM🤘) VERIFY IT (IN PROD) WATCH IT
- 28. DECIDE IT VERIFY IT (WFM🤘) VERIFY IT (IN PROD) WATCH IT BUILD IT
- 29. DECIDE IT VERIFY IT (WFM🤘) VERIFY IT (IN PROD) WATCH IT BUILD IT
- 30. DECIDE IT VERIFY IT (WFM🤘) VERIFY IT (IN PROD) WATCH IT BUILD IT
- 31. DECIDE IT VERIFY IT (WFM🤘) VERIFY IT (IN PROD) WATCH IT BUILD IT Like debug statements in production data
- 32. BUILD IT DECIDE IT VERIFY IT (WFM🤘) VERIFY IT (IN PROD) WATCH IT 🏁
- 33. BUILD IT DECIDE IT VERIFY IT (WFM🤘) VERIFY IT (IN PROD) WATCH IT 🏁 Feature flags and flexible observability tools = manual canarying … except we can do it for everything 👾
- 34. BUILD IT DECIDE IT VERIFY IT (WFM🤘) VERIFY IT (IN PROD) WATCH IT 🏁 👾
- 35. BUILD IT DECIDE IT VERIFY IT (WFM🤘) VERIFY IT (IN PROD) WATCH IT 🏁 👾
- 36. BUILD IT DECIDE IT VERIFY IT (WFM🤘) VERIFY IT (IN PROD) WATCH IT
- 37. BUILD IT DECIDE IT VERIFY IT (WFM🤘) VERIFY IT (IN PROD) WATCH IT
- 38. DECIDE IT VERIFY IT (WFM🤘) VERIFY IT (IN PROD) WATCH IT BUILD IT
- 39. BUILD ID FEATURE FLAGS CUSTOMER IDS
- 40. BUILD ID FEATURE FLAGS CUSTOMER IDS GRAPHS ALERTS CODE RELEASES
- 41. 1. HYPOTHESIS 2. INSTRUMENTATION (MAYBE) 3. VALIDATION (OR NOT) 4. ONWARD
- 42. TAKING THE FIRST FEW STEPS
- 43. CONCEPTUALLY ▸ Start at the edge with basic, common attributes (e.g. HTTP)
- 44. ▸ Start at the edge with basic, common attributes (e.g. HTTP) ▸ Business-relevant or infrastructure-specific characteristics (e.g. customer ID, DB replica set) CONCEPTUALLY
- 45. ▸ Start at the edge with basic, common attributes (e.g. HTTP) ▸ Business-relevant or infrastructure-specific characteristics (e.g. customer ID, DB replica set) ▸ Temporary additional fields for validating hypotheses CONCEPTUALLY
- 46. ▸ Start at the edge with basic, common attributes (e.g. HTTP) ▸ Business-relevant or infrastructure-specific characteristics (e.g. customer ID, DB replica set) ▸ Temporary additional fields for validating hypotheses ▸ Prune stale fields (if necessary) CONCEPTUALLY
- 47. ▸ Contextual, structured data SOME BEST PRACTICES
- 48. ▸ Contextual, structured data ▸ Common set of nouns and consistent naming SOME BEST PRACTICES
- 49. ▸ Contextual, structured data ▸ Common set of nouns and consistent naming ▸ Don't be dogmatic; let the use case dictate the ingest pattern SOME BEST PRACTICES
- 50. ▸ Contextual, structured data ▸ Common set of nouns and consistent naming ▸ Don't be dogmatic; let the use case dictate the ingest pattern ▸ e.g. instrumenting individual reads while batching writes SOME BEST PRACTICES
- 51. AN EXAMPLE SCHEMA EVOLUTION
- 52. first pass: - server_hostname - method - url - build_id - remote_addr - request_id - status - x_forwarded_for - error - event_time - team_id - payload_size - sample_rate then we added: - dropped - get_schema_dur_ms - protobuf_encoding_dur_ms - kafka_write_dur_ms - request_dur_ms - json_decoding_dur_ms +others a couple of days later, we added: - offset - kafka_topic - chosen_partition after that: - memory_inuse - num_goroutines a week after that: - warning - drop_reason and on and on, adding 2-3 fields every couple of weeks: - user_agent - unknown_columns - dataset_partitions - dataset_id - dataset_name - api_version - create_marker_dur_ms - marker_id - nil_value_for_columns - batch - gzipped - batch_datapoint_lens - batch_num_datasets - batch_process_datapoints_dur_ms - batch_validate_datasets_dur_ms - batch_dataset_names - dataset_columns - event_columns
- 53. ▸ Stop writing software based on intuition, start backing it up with data ▸ Teach observability tools to speak more than "Ops" ▸ ??? (← a.k.a., Ask lots of questions and validate hypotheses) ▸ Profit! DEVELOPERS, OUR MISSION: THANKS!@cyen