Observability of InfluxDB IOx: Tracing, Metrics and System Tables

Raphael Taylor-Davies
Observability of InﬂuxDB
IOx: tracing, metrics and
system tables

© 2021 InﬂuxData. All rights reserved.
2
Me
• Software engineer at InﬂuxData for ~6 months
• Lifecycle
• In-memory catalog
• APIs
• Previously worked as Platform/Infrastructure Engineer
• Working with Rust ~2 years

3
This Talk
• What is exposed
• Why is it useful
• How is it implemented

4
What

5
What: Management API/CLI
$ influxdb_iox database list
[“company_sensors”]
$ influxdb_iox database get company_sensors
{
"name": "company_sensors",
"lifecycleRules": {
"bufferSizeSoft": "42949672960",
"bufferSizeHard": "45097156608",
…
}
…
}

6
What: Management API/CLI
$ $ influxdb_iox operation list
[{
"id": 114328,
"wall_time": “844ms”,
"job": {
"CompactChunks": {
"partition": {
"db_name": "844910ece80be8bc_3c0bd4c89186ca89",
"table_name": "http_api_requests_total",
"partition_key": "2021-09-06 13:00:00"
},
"chunks": [371,372]
}
},
"status": "Success",
...
}, ...
]⏎

7
What: System Tables
$ influxdb_iox sql
> use company_sensors;
company_sensors> select partition_key, storage, memory_bytes, object_store_bytes, row_count from
system.chunks where table_name='http_request' limit 5;
+---------------------+-----------------+--------------+--------------------+-----------+
| partition_key | storage | memory_bytes | object_store_bytes | row_count |
+---------------------+-----------------+--------------+--------------------+-----------+
| 2021-08-23 15:00:00 | ObjectStoreOnly | 245693 | 52881438 | 5101475 |
+---------------------+-----------------+--------------+--------------------+-----------+

8
What: System Tables
$ influxdb_iox sql
> use company_sensors;
company_sensors> select table_name, sum(object_store_bytes) from system.chunks where table_name
like 'http_api%' group by table_name;
+-------------------------------------------------+-------------------------+
| table_name | SUM(object_store_bytes) |
+-------------------------------------------------+-------------------------+
| http_api_query_service_requests_total | 462729148 |
| http_api_query_service_requests_active | 32543301 |
| http_api_request_duration_seconds | 63534303368 |
| http_api_query_service_request_duration_seconds | 300583101 |
| http_api_requests_total | 7969265730 |
+-------------------------------------------------+-------------------------+

9
What: Metrics
$ http :8080/metrics
# HELP jemalloc_memstats_bytes jemalloc memstats
# TYPE jemalloc_memstats_bytes gauge
jemalloc_memstats_bytes{stat="active"} 55668879360
jemalloc_memstats_bytes{stat="alloc"} 52229651864
...
# HELP http_requests_total accumulated total requests
# TYPE http_requests_total counter
http_requests_total{path="/health",status="ok"} 337369
http_requests_total{path="/metrics",status="ok"} 33742
# HELP http_request_duration_seconds distribution of request latencies
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{path="/health",status="ok",le="0.005"} 337369
http_request_duration_seconds_bucket{path="/health",status="ok",le="+Inf"} 337369
http_request_duration_seconds_sum{path="/health",status="ok"} 0.16328641400000019
http_request_duration_seconds_count{path="/health",status="ok"} 337369
...

10
What: Logs
$ kubectl logs iox-0 iox
level=info msg="Chunk is getting sorted" chunk_type="MUB" chunk_ID=424 target="query::provider"
location="query/src/provider.rs:822" time=1630935055665806764
level=info msg="persisting partition as exceeds row threshold" db_name=company_sensors
partition="Partition('844910ece80be8bc_3c0bd4c89186ca89':'http_query_response_bytes':'2021-09-06
13:00:00') locked for read" persistable_row_count=5104599 target="lifecycle::policy"
location="lifecycle/src/policy.rs:369" time=1630935055673123030
level=info msg="Chunk is not yet sorted and will get sorted in build_sort_plan" chunk_type="MUB"
chunk_ID=428 target="query::provider" location="query/src/provider.rs:808"
time=1630935055688550768
...

11
What: Distributed Tracing
Ingress
Auth Service
Service A Service B
Context Propagation
Ingress: “/foo”
Auth: “/auth”
Ingress: isAuth()
SQL: “...”
Ingress: foo()
Service A: “/foo”
Service A: bar()
Service B: “/bar”
Time

12
What: Distributed Tracing

13
Why

14
Why: Scenario 1: Out Of Memory
Memory
Usage
Time
Jemalloc Usage
Chunk Usage

15
Why: Scenario 1: Out Of Memory
Memory
Usage
Time
Jemalloc Usage
Chunk Usage

16
Why: Scenario 2: Zombie Compaction
Number
of
Chunks
Time
Unpersisted Chunks
Persisted Chunks
Writes Stopped
~50 mins
~5 mins

17
$ kubectl logs iox-0 iox
level=info msg="persisting partition as exceeds row threshold"…
level=debug msg="no persistable windows or previous outstanding persist"…
…
TaskTracker { id: TaskId(48017), state: TrackerState { start_instant: Instant { tv_sec: 2580549,
tv_nsec: 345754677 }, cancel_token: CancellationToken { is_cancelled: false }, cpu_nanos:
140765855467, wall_nanos: 2786629479924, created_futures: 1, pending_futures: 0,
pending_registrations: 0, ok_futures: 1, err_futures: 0, cancelled_futures: 0, metadata:
PersistChunks { partition: PartitionAddr { db_name: "844910ece80be8bc_3c0bd4c89186ca89",
table_name: "storage_usage_bucket_cardinality", partition_key: "2021-08-26 09:00:00" }, chunks:
[2371, 2482, 2814, 2956, 2987, 3032, 3177, 3343, 3354, 3355, 3356, 3357, 3358] } }

18
Number
of
Chunks
Time
Unpersisted Chunks
Persisted Chunks
Writes Stopped
~50 mins
~5 mins

19
Why: Scenario 3: Overlapping Timestamps
$ select table_name,
partition_key,
row_count,
(cast(min_value as bigint) - cast(previous_max_value as bigint)) / 1000000000 as
delta_seconds
from (select *,
LAG(max_value)
OVER (partition by table_name, partition_key order by table_name, partition_key,
chunk_id) as previous_max_value
from system.chunk_columns
where column_name = 'time'
and storage = 'ReadBufferAndObjectStore' and partition_key like '2021-09-07%');

20
Why: Scenario 3: Overlapping Timestamps
+----------------------------------+---------------------+-----------+---------------+
| table_name | partition_key | row_count | delta_seconds |
+----------------------------------+---------------------+-----------+---------------+
| storage_usage_bucket_cardinality | 2021-09-07 06:00:00 | 118079869 | |
| storage_usage_bucket_cardinality | 2021-09-07 07:00:00 | 6119043 | 10 |
…

21
Why: Scenario 4: Query Performance
● Identiﬁcation
● Diagnosis

22
Why: Scenario 5: Ingest Latency
● Time to Bytes Readable (TTBR)
● Trace Propagation

23
How

24
How: Management API/CLI
● prost (protobuf)
● tonic (gRPC)
● pbjson (protobuf <-> JSON)
● clap + structopt (CLI)

25
How: System Tables
● Datafusion TableProvider

26
How: Metrics
● Open Telemetry
● Custom Crate
○ Metric abstraction
○ Prometheus

27
How: Logging
● tokio-tracing

28
How: Distributed Tracing
● tokio-tracing
● Custom Crate
○ tower middleware
○ context propagation
○ opentelemetry-jaeger

Observability of InfluxDB IOx: Tracing, Metrics and System Tables

More Related Content

What's hot (20)

Similar to Observability of InfluxDB IOx: Tracing, Metrics and System Tables (20)

More from InfluxData (20)

Recently uploaded (20)

Observability of InfluxDB IOx: Tracing, Metrics and System Tables