![Instrumentation – Add Debugging Information
22
Function AddToBasket(var productId, var quantity)
debug.AddParameters(“AddToBasket”, [[“ProductId”, productid],[“quantity”, quantity]])
metrics.Count(“AddToBasket”, 1)
start = time()
if (quantity < 0)
Log(“Error: Negative quantity value”)
metrics.Count(“AddToBasketFailure”, 1)
debug.AddError(“AddToBasket”, GetErrorData())
return false
var product = Dal.GetProductById(productId)
debug.AddValue(“AddToBasket”, [[“product”, product]])
metrics.Measure(“AddToBasket_GetProductById”, time() – start)
BasketService.Add(product, quantity)
metrics.Measure(“AddToBasket”, time() – start)
return true](https://image.slidesharecdn.com/observabilityandmore-architecturenext2020-200713004609/85/Observability-and-more-architecture-next-2020-22-320.jpg)
![Instrumentation – Original vs. Instrumented Code
23
Function AddToBasket(var productId, var quantity)
debug.AddParameters(“AddToBasket”, [[“ProductId”, productid],[“quantity”, quantity]])
metrics.Count(“AddToBasket”, 1)
start = time()
if (quantity < 0)
Log(“Error: Negative quantity value”)
metrics.Count(“AddToBasketFailure”, 1)
debug.AddError(“AddToBasket”, GetErrorData())
return false
var product = Dal.GetProductById(productId)
debug.AddValue(“AddToBasket”, [[“product”, product]])
metrics.Measure(“AddToBasket_GetProductById”, time() – start)
BasketService.Add(product, quantity)
metrics.Measure(“AddToBasket”, time() – start)
return true](https://image.slidesharecdn.com/observabilityandmore-architecturenext2020-200713004609/85/Observability-and-more-architecture-next-2020-23-320.jpg)
![Instrumentation – Call Context
26
Function AddToBasket(var productId, var quantity, var context)
debug.AddParameters(context, “AddToBasket”, [[“ProductId”, productid],[“quantity”, quantity]])
metrics.Count(context, “AddToBasket”, 1)
start = time()
if (quantity < 0)
Log(context, “Error: Negative quantity value”)
metrics.Count(context, “AddToBasketFailure”, 1)
debug.AddError(context, “AddToBasket”, GetErrorData())
return false
var product = Dal.GetProductById(context, productId)
debug.AddValue(context, “AddToBasket”, [[“product”, product]])
metrics.Measure(context, “AddToBasket_GetProductById”, time() – start)
BasketService.Add(context, product, quantity)
metrics.Measure(context, “AddToBasket”, time() – start)
return true
Context:
Call Id
URL
HTTP Method
DB Host
User Info
Timing Info](https://image.slidesharecdn.com/observabilityandmore-architecturenext2020-200713004609/85/Observability-and-more-architecture-next-2020-26-320.jpg)
Observability and more architecture next 2020
- 1. Observability & More Alon Fliess Chief Architect [email protected] @alon_fliess http://alonfliess.me http://codevalue.net
- 2. Cloudflare blames ‘bad software’ deployment for today’s outage
- 3. About Me Alon Fliess: Chief Software Architect & Co-Founder at OzCode & CodeValue More than 30 years of hands-on experience Microsoft Regional Director & Microsoft Azure MVP Spend most of my time in project analysis, architecture, design Code at night
- 4. Azure Israel https://www.meetup.com/AzureIsrael 4
- 5. Agenda DevOps, the true story Microservice Architecture, the complexity shift Ops & Monitoring Site Reliable Managers Developers & Observability Business (marketing, sales, management) and observability Application Performance Monitoring How does it work? Distributed Tracing Production problem solving 5
- 6. The Essence of DevOps Better Software, Faster! When Development and Operations Synergize Covers the *entire* Application Lifecycle 6
- 7. Microservice Architecture == Complexity Shift 7
- 8. Ops Vital Signs: Heartbeat, Blood Pressure, Temperature 8
- 9. What Do Site Reliability Managers (SRE) Want? 9
- 10. What Do Developers Want? 10
- 11. What Do Marketing & Sales Teams Want? 11
- 12. What is Observability? (Twitter 2013) 12
- 13. Gartner Critical Capabilities for APM (May 2019) 13 Business Analysis Anomaly Detection IT Operations DevOps Release Application Support Application Development Application Owner Use Cases
- 14. 14
- 15. APM Players Dynatrace AppDynamics (Cisco) Datadog Splunk Broadcom (CA Technologies) New Relic Riverbed IBM Instana Oracle Tingyun SolarWinds ManageEngine Micro Focus 15
- 16. How Does Monitoring & Tracing Work? 16 Operating Systems APM system tracking agent installed on the machine CPU, Memory, I/O, Network Code Tracing Instrumentation Manual Auto Runtime data collection
- 17. Instrumentation – Original Pseudo Code 17 Function AddToBasket(var productId, var quantity) if (quantity < 0) return false var product = Dal.GetProductById(productId) BasketService.Add(product, quantity) return true
- 18. Instrumentation – Add Logging on Errors 18 Function AddToBasket(var productId, var quantity) if (quantity < 0) Log(“Error: Negative quantity value”) return false var product = Dal.GetProductById(productId) BasketService.Add(product, quantity) return true
- 19. Instrumentation – Add Metrics of Usage and Errors 19 Function AddToBasket(var productId, var quantity) metrics.Count(“AddToBasket”, 1) if (quantity < 0) Log(“Error: Negative quantity value”) metrics.Count(“AddToBasketFailure”, 1) return false var product = Dal.GetProductById(productId) BasketService.Add(product, quantity) return true
- 20. Instrumentation – Measure Latency 20 Function AddToBasket(var productId, var quantity) metrics.Count(“AddToBasket”, 1) start = time() if (quantity < 0) Log(“Error: Negative quantity value”) metrics.Count(“AddToBasketFailure”, 1) return false var product = Dal.GetProductById(productId); BasketService.Add(product, quantity); metrics.Measure(“AddToBasket”, time() – start); return true;
- 21. Instrumentation – Measure Latency Everywhere 21 Function AddToBasket(var productId, var quantity) metrics.Count(“AddToBasket”, 1) start = time() if (quantity < 0) Log(“Error: Negative quantity value”) metrics.Count(“AddToBasketFailure”, 1) return false var product = Dal.GetProductById(productId) metrics.Measure(“AddToBasket_GetProductById”, time() – start) BasketService.Add(product, quantity) metrics.Measure(“AddToBasket”, time() – start) return true
- 22. Instrumentation – Add Debugging Information 22 Function AddToBasket(var productId, var quantity) debug.AddParameters(“AddToBasket”, [[“ProductId”, productid],[“quantity”, quantity]]) metrics.Count(“AddToBasket”, 1) start = time() if (quantity < 0) Log(“Error: Negative quantity value”) metrics.Count(“AddToBasketFailure”, 1) debug.AddError(“AddToBasket”, GetErrorData()) return false var product = Dal.GetProductById(productId) debug.AddValue(“AddToBasket”, [[“product”, product]]) metrics.Measure(“AddToBasket_GetProductById”, time() – start) BasketService.Add(product, quantity) metrics.Measure(“AddToBasket”, time() – start) return true
- 23. Instrumentation – Original vs. Instrumented Code 23 Function AddToBasket(var productId, var quantity) debug.AddParameters(“AddToBasket”, [[“ProductId”, productid],[“quantity”, quantity]]) metrics.Count(“AddToBasket”, 1) start = time() if (quantity < 0) Log(“Error: Negative quantity value”) metrics.Count(“AddToBasketFailure”, 1) debug.AddError(“AddToBasket”, GetErrorData()) return false var product = Dal.GetProductById(productId) debug.AddValue(“AddToBasket”, [[“product”, product]]) metrics.Measure(“AddToBasket_GetProductById”, time() – start) BasketService.Add(product, quantity) metrics.Measure(“AddToBasket”, time() – start) return true
- 24. Instrumentation and Tracing Automation Aspect Oriented Approach Communication level instrumentation Pipeline interception – technology depended Resource performance counters – DB statistics for example Code Instrumentation Manual – deploy a package and call it Automatic – bytecode instrumentation libraries and tools Distributed Tracing Passing call context between services 24
- 25. Distributed Tracing 25 Id:123 Application A Service A B Service B Span Span Span
- 26. Instrumentation – Call Context 26 Function AddToBasket(var productId, var quantity, var context) debug.AddParameters(context, “AddToBasket”, [[“ProductId”, productid],[“quantity”, quantity]]) metrics.Count(context, “AddToBasket”, 1) start = time() if (quantity < 0) Log(context, “Error: Negative quantity value”) metrics.Count(context, “AddToBasketFailure”, 1) debug.AddError(context, “AddToBasket”, GetErrorData()) return false var product = Dal.GetProductById(context, productId) debug.AddValue(context, “AddToBasket”, [[“product”, product]]) metrics.Measure(context, “AddToBasket_GetProductById”, time() – start) BasketService.Add(context, product, quantity) metrics.Measure(context, “AddToBasket”, time() – start) return true Context: Call Id URL HTTP Method DB Host User Info Timing Info
- 27. Instrumentation – Using Span 27 Function AddToBasket(var productId, var quantity, var context) span = trace.BeginSpan(context, {“AddToBasket”, productid, quantity}) if (quantity < 0) span.Error(“Negative quantity value”) return false; var product = Dal.GetProductById(context, productId) span.AddValue(“product”, product) BasketService.Add(context, product, quantity) span.End() return true; Span: Call Id URL HTTP Method DB Host User Info Timing Info
- 29. What Do SREs & Developers Want – From Each Other? 29
- 30. New Relic APM Dashboard
- 31. APM Error Analysis – Not Enough Information Error Rate Request information Stack trace APM systems can assist in health monitoring and fault first aid
- 32. Production Problem Solving Challenges 10kg Can’t mess with data 10kg No Debugging tools 10kg Code is optimized 10kg Older source code version 10kg Can’t impact performance 10kg Data must stay in a secure env. 10kg Data is private and contains PII 10kg Very hard to reproduce the bug
- 33. Problem Solving With APM 33
- 34. Production Problem Solving Platforms OzCode OverOps Rookout Application Insights 34
- 35. Problem Solving With a Production Debugger 35
- 37. Summary 37
- 38. Q A 38
Editor's Notes
- #8: MSA – many small parts deployed and communicate Simple components, Complex combination Very hard to follow a request that spans many services Must have automation process to overcome the complexity Must have health monitor, performance monitor and cross-services error handling TOOLS!!!
- #9: More than CI/CD Ops First aid medic, take vital signs CPU, Network, IO, Memory Request throughput and latency
- #10: Wants easy life. Eats the meal that the Dev team cooked. The customer of the Dev team Bugs, Problem Solving Need to know the current situation with the current problems For example, can role back to a previous version, but need to know the status of the bug fix
- #11: Information, Debuggability Reproduce the problem
- #12: Analytics Business Insights Usage
- #13: As Twitter has moved from a monolithic to a distributed architecture, our scalability has increased dramatically. Zipking – a distributed tracing system (https://zipkin.io/)
- #14: Business Analysis - business related KPI IT Service Monitoring - health of Key Services Root Cause Analysis - a failure or degradation Anomaly Detection identifying system observations that do not conform to an expected behavior Distributed Profiling track transactions across a mesh of interconnected nodes, followed by detection of where along the path the degradation appears to be happening Application Debugging production debugging capabilities, based on distributed date collection
- #20: Enable saying: 15% of our request fails
- #26: Errors & problems root cause may be the result of
- #27: Problem happens only with a specific user or URLs
- #28: Problem happens only with a specific user or URLs
- #29: OpenTelemetry makes robust, portable telemetry a built-in feature of cloud-native software. OpenTelemetry provides a single set of APIs, libraries, agents, and collector services to capture distributed traces and metrics from your application. You can analyze them using Prometheus, Jaeger, and other observability tools.
- #38: DevOps, the true story Microservice Architecture, the complexity shift Ops & Monitoring Site Reliable Managers Developers & Observability Business (marketing, sales, management) and observability Application Performance Monitoring How does it work? Distributed Tracing Production problem solving