Observability, Distributed Tracing, and Open Source: The Missing Primer
1 like896 views
Open source tools like OpenTelemetry, OpenTracing, and W3C Trace Context are helping to standardize distributed tracing and observability. This allows developers to understand problems in microservices architectures by propagating unique trace IDs and collecting metrics and traces across services. While open source tools are useful for development and pre-production, commercial solutions are needed to handle production workloads at scale with additional features like access control and automated instrumentation. Standardization through open source is key to managing today's complexity in distributed systems.
Observability, Distributed Tracing, and Open Source: The Missing Primer
- 1. Observability, Distributed Tracing, and Open Source The Missing Primer
- 3. 3
- 4. 4
- 5. 5 • DanielKhan [email protected] @dkhan • Dir. TechnologyStrategy @Dynatrace • Everything Open Source Monitoring & standards& our contributionsto it • Chairof W3C Trace Context About me
- 6. 6 Why I am doing this talk Distributed Tracing Observability W3C Trace Context OpenCensus OpenTracing OpenTelemetry Metrics Span Trace
- 7. 7 Application In the Beginning there was the Monolith Presentation Business Logic Data Access Database Services Presentation API Gateway Auth Inventory CartAccount Offers Shipping CheckoutStatus Wire
- 8. 8 Developmentin a Microservices World Cart Dev Preproduction Cart Auth InventoryAccount Offers Shopping CheckoutStatus Push Cart • Latency • Response Time • Error Rate • Number of queries KPI’s
- 9. 9 Metrics Source: https://techblog.commercetools.com/adding-consistency-and-automation-to-grafana-e99eb374fe40 … containtime correlated datapoints • Counter Monotonously increasing values Think: Odometer • Gauge Increasing and decreasing values Think: Tachometer • Histogram Groups values into buckets Think: Knock events 0-50mph, 51-100mph, …
- 10. 10 Collecting and Charting Metrics
- 12. 12 Complecity has movedto the NetworkLayer Client API GW Service Service Service Service Service ServiceCart Which requests lead to an error in our cart service? Trace a42b a42b a42b a42b a42b a42b a42b a42b a42b = Trace Context
- 13. 13 A Trace is a Tree of Spans Trace Span Span Span Click GW API Spans represent a single operationand containmetadatalike the HTTP method, or a databasequery, or an error code JDBC Span: callDB() Span: JDBC call
- 14. 14 Trace Context Propagation Cartsa42b a42b Extract Inject In process propagation Auto Instrumentation • Zipkin • Sleuth • OpenTelemetry • Commercial • …
- 15. 15 Trace Context Header Formats • Proprietary • B3 Header (Zipkin) • W3C Trace Context What’sthe header name and what does it contain?
- 16. 16 W3C Trace Context Service A API GW Service B Trace Service C OpenTelemetry AWS Zipkin OpenTelemetry Goal: All monitoring systems and middlewaresagree on one format for trace context propagation Span Span Span Span
- 17. 17 W3C Trace Context Format traceparent: 00-0af7651916cd43dd8448eb211c80319c-00f067aa0ba902b7-01 tracestate: rojo=00f067aa0ba902b7,congo=t61rcWkgMzE Version TraceID ParentID Flags
- 18. 18 Data collection So far we just instrumentedthe code to propagatecontext but no data has been collected Trace Span Span Span Agent Agent Agent Click GW API TraceContext TraceContext MonitoringSystem Storage
- 19. 19 Data Collection & PresentationSystems Solution Agents Instrumentation Storage Presentation Zipkin / Sleuth + + + + Jaeger - - + + OpenTelemetry + + - - Commercial + + + +
- 20. 20 Zipkin
- 21. 21 Jaeger
- 22. 22 Commercial
- 23. 23 Entity Model Based Service Flow
- 25. 25 Solving our Cart Problem Client API GW Service Service Service Service Service ServiceCart Trace Client Service Currency Cart API GW GET: Currency=EURO
- 26. 26 What we did to Solve the Problem 1. We used metrics to learn about a problem 2. We used distributedtracing to pass along a unique ID per trace • For that,we used auto instrumentationto extract and inject the trace ID 3. We used a monitoring system and its agentsto collected traces and we could filter transactionsthatproduced an error 4. We looked into the metadata of such a transactionto identify how it differs from succeeding ones
- 27. 27 You’ve mentionedOpenTelemetry … + = In early 2019 OpenCensusand OpenTracingmerged into OpenTelemetry Metrics, Traces, Logs
- 28. 28 APIs SDKs Exporters Collector
- 29. 29
- 30. 30
- 31. 31
- 32. 32
- 33. 33 OpenTelemetry – Developer usecases • Cloud nativemicroservices architectures are hard to trace and debug during development • In developmentOpenTelemetry can be used to either • manuallycreate spans to trace certain execution paths • use provided auto-instrumentation tomake a system observable • As backend and UI, Jaeger is the most popular tool. It’s open source and solely displaystraces
- 34. 34 OpenTelemetry – in Production • Provides just a fraction of what modern tools provide • Traces • Metrics • Logs • Topology • Behavior • Code level visibility • Metadata • Manual instrumentation codeneeds to be kept up-to-date • A backend needs to be maintained • No support model if instrumentation breaksproductioncode • No enterprise features (access control, throttling, scaling, …)
- 35. 35 Why do Vendors Care then?
- 36. 36 OpenTelemetry Company Contribution Stats Google Microsoft Dynatrace
- 37. 37
- 38. 38 What happens when we add support for a new framework? • Today, our engineers reverse engineer frameworks to add instrumentationsupport to them • Every time an update is released, the instrumentationcode is being tested. • In case of issues, it goes back to the developmentteam who needs to fix it and deploy an update. • The whole process is automated and transparent to the customer ☺ • This is costly and time consuming
- 39. 39 In-process tracing Click GW API MonitoringSystem Trace Span Span Span HZQ Span: doHZQ()() Span: HZQ call OTEL HZQ Wrapper
- 40. 40 “We want every platform and library to be pre-instrumented with OpenTelemetry and we’re committed to making this as easy as possible.” Sergey Kanzhelev (Google)
- 41. 41 What is Observability and how does it differ fromMonitoring? 1. In control theory, observability is a measure of how well internal states of a system can be inferred from knowledgeof its external outputs. Source: Wikipedia 2. In software development, observabilityisachieved by adding code (instrumentation)that emits telemetry data. 3. Monitoringis the act of displayingand analyzing this telemetry data. 4. Monitoringalone can tell you that there is a problem. E.g. ”We see that some users experience a 50% higher response time on check-out” 5. Observablityhelps finding the root cause (the why) by providingdatathat can be correlatedand analysed freely even if this problem is completelynew to you (unknown unknowns) E.g. “The response time of the checkout increases exponentially with the number of items in the basket, because of a misplaced for loop that executes the same database query times the number of items for every item in the basket”
- 42. 42 Putting it all Together • Metrics can help you to learn that there is a problem • Distributedtracing becomes increasingly importantto understandmulti-tier execution paths and root causes of problems • Developersnow rely on metrics and traces to understandhow their service functionsin their microservice architectures • Pure Open Source solutionsare viable for pre-prod environments • Standardization isthe only way to tackle today’scomplexity and Open Source is the key driver • Vendorsare prepared to tap intodata collected by Open Source standard toolsto add enterprise features on top to support web-scale workloads