Grafana Distributed Tracing Example with: Tempo, Prometheus, Loki, Grafana and Python Flask
This project showcases microservices architecture with distributed tracing powered by OpenTelemetry, integrated with Grafana Tempo. It also leverages derived fields in Grafana Loki to seamlessly link trace IDs to Tempo, providing a unified and smooth experience for viewing traces directly from logs.
The request flow will look like the following:
Boot the stack:
make upThe datasources will be auto-configured defined in configs/grafana/datasources.yaml.
Run the request in ./create_order.sh:
./create_order.shThe response will be similar to the following:
{
"message": "Order 119 created and payment authorized",
"status": "success",
"trace_id": "25fb067b54465ea4ccecea93694c8824"
}Logs using make logs:
api-gateway | [2024-08-11 20:28:42,387] DEBUG in app: api-gateway received post request
api-gateway | [2024-08-11 20:28:42,388] DEBUG in app: api-gateway makes a request to order-service trace_id=b6512704d84b6892c72f4b0706ba6d06
order-service | [2024-08-11 20:28:42,397] DEBUG in app: order-service received a post request
order-service | [2024-08-11 20:28:42,398] DEBUG in app: order-service makes a post request to inventory-service trace_id=b6512704d84b6892c72f4b0706ba6d06
inventory-service | [2024-08-11 20:28:42,408] DEBUG in app: inventory-service received a post request
inventory-service | [2024-08-11 20:28:42,409] DEBUG in app: inventory-service about to make a database query. trace_id=b6512704d84b6892c72f4b0706ba6d06
inventory-service | [2024-08-11 20:28:42,424] DEBUG in app: the availability in the inv db is 96
warehouse-service | [2024-08-11 20:28:42,434] DEBUG in app: warehouse-service received a post request
warehouse-service | [2024-08-11 20:28:42,435] INFO in app: warehouse-service about to make a database query. trace_id=b6512704d84b6892c72f4b0706ba6d06
warehouse-service | 192.168.128.8 - - [11/Aug/2024 20:28:42] "POST /warehouse/reserve HTTP/1.1" 200 -
inventory-service | 192.168.128.12 - - [11/Aug/2024 20:28:42] "POST /inventory/check HTTP/1.1" 200 -
payment-service | [2024-08-11 20:28:42,469] INFO in app: payment-service logged trace_id=b6512704d84b6892c72f4b0706ba6d06
fraud-service | [2024-08-11 20:28:42,479] INFO in app: fraud-service logged trace_id=b6512704d84b6892c72f4b0706ba6d06
fraud-service | [2024-08-11 20:28:42,480] INFO in app: Transaction passed fraud check: order_id=754, user_id=123
fraud-service | 192.168.128.9 - - [11/Aug/2024 20:28:42] "POST /fraud/check HTTP/1.1" 200 -
payment-service | 192.168.128.12 - - [11/Aug/2024 20:28:42] "POST /payment/authorize HTTP/1.1" 200 -
order-service | 192.168.128.13 - - [11/Aug/2024 20:28:42] "POST /order HTTP/1.1" 200 -
api-gateway | 192.168.128.1 - - [11/Aug/2024 20:28:42] "POST /api/order HTTP/1.1" 200 -Explore traces:
Query our cluster and Table view:
Node Graph:
When we use the order-service container logs in Loki:
From the response of our request, we get a trace_id, when we use TraceQL in Explore to search for our trace id and we can see all the spans from the given trace:
Since we are using remote_write in Tempo, we are writing metrics from Tempo to Prometheus, and we can access metrics such as http request rate from client to server:
Similarly, we can access the p90 percentile request latencies of requests using the request path /api/order:
With span.set_attribute we can enrich some of the visuals:
with tracer.start_as_current_span("query_inventory_database") as span:
inventory_item = Inventory.query.filter_by(id=item_id).first()
sql_query = f"SELECT * FROM inventory WHERE item_id = {item_id}"
span.set_attribute("db.query", sql_query)
span.set_attribute("inventory.item_id", item_id)
span.set_attribute("inventory.requested_quantity", quantity)
span.set_attribute("inventory.availability", inventory_item.availability)Produces:
