Custom Distributed Tracing in Azure Functions (2021-02-27)
0 likes518 views
The document presents an overview of custom distributed tracing and observability practices in Azure Functions discussed during a global integration bootcamp in 2021. It emphasizes the importance of effective troubleshooting in distributed systems, particularly with backend services, and outlines common operational requirements along with a proposed structured logging approach and correlation identifiers for message tracing. Additionally, it includes a sequence diagram, solution components, and resources for further learning.
Custom Distributed Tracing in Azure Functions (2021-02-27)
- 1. @pacodelacruz Paco de la Cruz @ Melbourne AU Custom Distributed Tracing & Observability Practices in Azure Functions Global Integration Bootcamp 2021 @pacodelacruz
- 3. @pacodelacruz Introduction Scenario Common operation requirements Tracing design: spans, key-value pairs Solution design: sequence diagram, components Demo Q & A Agenda
- 4. @pacodelacruz Devs tend to focus on building and shipping services The solution in production becomes someone else’s problem Distributed systems will fail We need to give Ops teams the means to troubleshooting Particularly relevant with backend services running in the background Introduction – Why are we talking about this?
- 5. @pacodelacruz Scenario - Participants Webhook pushing user update events (batch) Publisher Azure Function Subscriber Azure Function Service Bus Queue Target System Publish–Subscribe solution of user update events from a HR system via a webhook (batch of events) to a bespoke target system that can’t receive webhook calls Integration layer with temporal decoupling via a queue
- 6. @pacodelacruz Opinionated approach Based on many years of experience Publish-Subscribe pattern Splitter pattern Approach could be tailored Scenario - Notes
- 7. @pacodelacruz Scenario – Sequence Diagram @pacodelacruz
- 8. @pacodelacruz Scenario – Sequence Diagram @pacodelacruz
- 9. @pacodelacruz Scenario – Sequence Diagram @pacodelacruz
- 10. @pacodelacruz Common Operations Requirements Requirement Potential Solution Ability to inspect incoming request payloads for troubleshooting purposes. Archive all request payloads as received. Ability to correlate individual messages to the original batch and understand whether all messages in a batch were processed successfully. Capture a batch record count and a batch identifier in all tracing events for individual messages. Ability to correlate all tracing events for every individual message. Capture a tracing correlation identifier in all tracing events for every message. Being able to find tracing events for a particular message. Capture business-related metadata in all relevant tracing logs, including an interface identifier, message type, and an entity identifier. Receive alerts when certain failures occur. Capture granular tracing event identifiers and tracing event statuses. Ability to differentiate transient failures. Capture a delivery count when messages are being retried to identify whether additional attempts are expected. Ability to query, filter, and troubleshoot what happens to individual messages. Enable correlation of tracing events with standard tracing checkpoints, granular event identifiers, status, and business-related metadata.
- 11. @pacodelacruz Tracing Spans “The span is the primary building block of a distributed trace, representing an individual unit of work done in a distributed system. Each component of the distributed system contributes a span - a named, timed operation representing a piece of the workflow.” - opentracing.io
- 12. @pacodelacruz Structured Logging Key-Value Pairs Key Description Scope BatchId Batch identifier to correlate individual messages to the original batch. It is highly recommended when using the splitter pattern. Cross-span CorrelationId Tracing correlation identifier of an individual message. Cross-span EntityType Business identifier of the message type being processed. This allows to filter or query tracing events for a particular entity type. E.g. UserEvent, PurchaseOrder, Invoice, etc. Cross-span EntityId Business identifier of the entity in the message. This together with the EntityType key-value pair allow to filter or query tracing events for messages related to a particular entity. E.g. UserId, PurchaseOrderNumber, InvoiceNumber, etc. Cross-span InterfaceId Business identifier of the interface. This allows to filter or query tracing events for a particular interface. Useful when an organisation defines identifiers for their integration interfaces. Span RecordCount Optional. Only applicable to batch events. Captures the number of individual messages or records that are present in the batch. Span DeliveryCount Optional. Only applicable to subscriber events of individual messages. Captures the number of times the message has been attempted to be delivered. It relies on the Service Bus message DeliveryCount property. Span LogLevel LogLevel as defined by Microsoft.Extensions.Logging Log SpanCheckpoint Defines the tracing span and whether it is the start or finish of it, e.g. PublisherStart or PublisherFinish. Having standard checkpoints allows correlating tracing events in a standard way. Log EventId Captures a specific tracing event that helps to query, analyse, and troubleshoot the solution with granularity. Log Status Stores the status of the tracing event, e.g., succeeded or failed Log
- 13. @pacodelacruz Solution Components Application Insights Tracing Logs Blob Storage Archive Webhook pushing user update events (batch) Publisher Azure Function Subscriber Azure Function Service Bus Queue Target System
- 25. @pacodelacruz Code walkthrough Components overview Call the publisher function Monitor the solution Query the traces Demo github.com/pacodelacruz/ observability-pubsub-functions
- 26. @pacodelacruz Costs associated with Application Insights (ingestion & retention) Tracing data can be client-side sampled or filtered Traces can be server-side sampled or throttled (max req/s & daily cap) Telemetry channel is async & best-efforts (Traces could be lost: app crashes, network issues, etc.) This approach must not be used for auditing or billing purposes Considerations
- 28. @pacodelacruz Additional Resources Intro https://platform.deloitte.com.au/articles/ distributed-tracing-and-observability-with-azure-functions-1-intro Design https://platform.deloitte.com.au/articles/ distributed-tracing-and-observability-with-azure-functions-2-design Implementation https://platform.deloitte.com.au/articles/ distributed-tracing-and-observability-with-azure-functions-3- implementation Repo github.com/pacodelacruz/observability-pubsub-functions