Infrastructure Monitoring with Postgres

Infrastructure Monitoring
(with Postgres, obviously)
Steve Simpson
StackHPC
steve@stackhpc.com
www.stackhpc.com

Overview
1) Background
2) Monitoring
Postgres for Metrics
3) Requirements
4) Data & Queries
5) Optimisation
Postgres for ...
6) Log Searching
7) Log Parsing
8) Queueing

Background
Systems Software Engineer
C, C++, Python

Background
Based in Bristol, UK
Thriving Tech Industry

Background
● Gnodal
● 10GbE Ethernet
● ASIC Verification
● Embedded Firmware
● JustOne Database
● Agile “Big Data” RMDBS
● Based on PostgreSQL
● Storage Team Lead

Background
Consultancy for HPC on OpenStack
Multi-tenant massively parallel workloads
Monitoring complex infrastructure
Stack
HPC

Background
Cloud orchestration platform
IaaS through API and dashboard
Multi-tenancy throughout
Network, Compute, Storage

Background
Operational visibility is critical
OpenStack is a complex, distributed application
…to run your complex, distributed applications

Monitoring Requirements
Gain visibility into the operation of the
hardware and software
e.g. web site, database, cluster, disk drive

Fault finding and alerting
Notify me when a server or service is
unavailable, a disk needs replacing, ...
Fault post-mortem, pre-emption
Why did the outage occur and what can we
do to prevent it next time

Utilisation and efficiency analysis
Is all the hardware we own being used?
Is it being used efficiently?
Performance monitoring and profiling
How long are my web/database requests?

Auditing (security, billing)
Tracking users use of system
Auditing access to systems or resources
Decision making, future planning
What is expected growth in data, or users?
What of the current system is most used?

Existing Tools
Checking and Alerting
Agents check on machines or services
Report centrally, notify users via dashboard
Store history of events in database

Existing Tools
Nagios / Icinga
ping -c 1 $host || mail -s “Help!” $me

Existing Tools
Kibana (+Elasticsearch/Logstash)

Existing Tools
Metrics
Periodically collect metrics, e.g. CPU%
Store in central database for visualization
Some systems allow checking on top

Existing Tools
Ganglia
Collector (gmond) + Aggregator (gmetad)

Existing Tools
https://ganglia.wikimedia.org/

Existing Tools
Grafana - visualization only

Existing Tools
Metrics Databases
● Ganglia (RRDtool)
● Graphite (Whisper)
● OpenTSDB (HBase)
● KairosDB (Cassandra)
● InfluxDB
● Prometheus
● Gnocchi
● Atlas
● Heroic
● Hawkular (Cassandra)
● MetricTank (Cassandra)
● Riak TS (Riak)
● Blueflood (Cassandra)
● DalmatinerDB
● Druid
● BTrDB
● Warp 10 (Hbase)
● Tgres (PostgreSQL!)

Existing Tools
Metrics Databases
● Ganglia [Berkley]
● Graphite [Orbitz]
● OpenTSDB [Stubleupon]
● KairosDB
● InfluxDB
● Prometheus [SoundCloud]
● Gnocchi [OpenStack]
● Atlas [Netflix]
● Heroic [Spotify]
● Hawkular [Redhat]
● MetricTank [Raintank]
● Riak TS [Basho]
● Blueflood [Rackspace]
● DalmatinerDB
● Druid
● BTrDB
● Warp 10
● Tgres

Existing Tools
2000
2010
2013 - 2015

Existing Tools
Software
Network
Storage
Servers

Existing Tools
Software
Network
Storage
Servers
Metrics
Logs

Monasca
Existing Tools
Software
Network
Storage Log API
Metric API
Alerting
Servers
Metrics
Logs

Monasca
Existing Tools
Software
Network
Storage Log API
Metric API
Alerting
MySQL
Servers
Metrics
Logs

Monasca
Existing Tools
Software
Network
Storage Log API
InfluxDB
Metric API
Alerting
MySQL
Servers
Metrics
Logs

Monasca
Existing Tools
Software
Network
Storage Log API
InfluxDB
Metric API
Alerting
Grafana
MySQL
SQLite
Servers
Metrics
Logs

Monasca
Existing Tools
Software
Network
Storage Log API
Logstash Elastic
InfluxDB
Metric API
Alerting
Grafana
MySQL
SQLite
Servers
Metrics
Logs

Monasca
Existing Tools
Software
Network
Storage Log API
Logstash Elastic
InfluxDB
Metric API
Alerting
Grafana
Kibana
MySQL
SQLite
Servers
Metrics
Logs

Monasca
Existing Tools
Software
Network
Storage Log API
Kafka
Logstash Elastic
InfluxDB
Metric API
Alerting
Grafana
Kibana
MySQL
SQLite
Servers
Metrics
Logs

Monasca
Existing Tools
Software
Network
Storage Log API
Kafka
Logstash Elastic
InfluxDB
Metric API
Alerting
Grafana
Kibana
MySQL
SQLite
Servers
Metrics
Logs
Zookeeper

Existing Tools
Commendable “right tool for the job” attitude, but…
How about Postgres?
Fewer points of failure
Fewer places to backup
Fewer redundancy protocols
One set of consistent data semantics
Re-use existing operational knowledge

Monasca
Existing Tools
Software
Network
Storage Log API
Kafka
Logstash Elastic
InfluxDB
Metric API
Alerting
Grafana
Kibana
SQLite
Servers
Metrics
Logs
Zookeeper

Monasca
Existing Tools
Software
Network
Storage Log API
Kafka
Logstash Elastic
InfluxDB
Metric API
Alerting
Grafana
Kibana
Servers
Metrics
Logs
Zookeeper

Monasca
Existing Tools
Software
Network
Storage Log API
Kafka
Logstash Elastic
Metric API
Alerting
Grafana
Kibana
Servers
Metrics
Logs
Zookeeper

Monasca
Existing Tools
Software
Network
Storage Log API
Kafka
Logstash
Metric API
Alerting
Grafana
Grafana?
Servers
Metrics
Logs
Zookeeper

Monasca
Existing Tools
Software
Network
Storage Log API
Logstash
Metric API
Alerting
Grafana
Grafana?
Servers
Metrics
Logs
Zookeeper

Monasca
Existing Tools
Software
Network
Storage Log API
Logstash
Metric API
Alerting
Grafana
Grafana?
Servers
Metrics
Logs

Monasca
Existing Tools
Software
Network
Storage Log API
Metric API
Alerting
Grafana
Grafana?
Servers
Metrics
Logs

Monasca
Existing Tools
Software
Network
Storage Log API
Metric API
Alerting
Grafana
Servers
Metrics
Logs

Requirements

Requirements
● ~45M values/day
(80x196 per 30s)
● 6 month history
● <1TB disk footprint
● <100ms queries

Combine Series
average over all
for {series=cpu}
[time range/interval]
Read Series
for each {type}
for {series=cpu}
[time range/interval]

List
Dimension
Values
List
Dimension
Names
List
Metric
Names
"metrics": [
"cpu.percent",
"cpu.user_perc",
"net.out_bytes_sec",
"net.out_errors_sec",
"net.in_bytes_sec",
"net.in_errors_sec"
…
]
"dimensions": [
"device",
"hostname",
"instance",
"mount_point",
"process_name",
"process_user"
…
]
"hostname": [
"dev-01",
"dev-02",
"staging-01",
"staging-02",
"prod-01",
"prod-02"
…
]

Data & Queries

"metric": {
"timestamp": 1232141412,
"name": "cpu.percent",
"value": 42,
"dimensions": { "hostname": "dev-01" },
"value_meta": { … }
}
JSON Ingest Format
Known, well defined structure
Varying set of dimensions key/values

CREATE TABLE measurements (
timestamp TIMESTAMPTZ,
name VARCHAR,
value FLOAT8,
dimensions JSONB,
value_meta JSON
);
Basic Denormalised Schema
Straightforward mapping onto input data
Data model for all schemas

SELECT
TIME_ROUND(timestamp, 60) AS timestamp,
AVG(value) AS avg
FROM
measurements
WHERE
timestamp BETWEEN '2015-01-01Z00:00:00'
AND '2015-01-01Z01:00:00'
AND name = 'cpu.percent'
AND dimensions @> '{"hostname": "dev-01"}'::JSONB
GROUP BY
timestamp
Single Series Query
One hour window | Single hostname
Measurements every 60 second interval

SELECT
AVG(value) AS avg,
dimensions ->> 'hostname' AS hostname
FROM
measurements
WHERE
AND '2015-01-01Z01:00:00'
GROUP BY
timestamp, hostname
Group Multi-Series Query
One hour window | Every hostname

SELECT
AVG(value) AS avg
FROM
measurements
WHERE
AND '2015-01-01Z01:00:00'
GROUP BY
timestamp
All Multi-Series Query
One hour window | Every hostname

SELECT DISTINCT
name
FROM
measurements
Metric Name List Query
:)

SELECT DISTINCT
JSONB_OBJECT_KEYS(dimensions)
AS d_name
WHERE
name = 'cpu.percent'
FROM
measurements
Dimension Name List Query
(for specific metric)

SELECT DISTINCT
dimensions ->> 'hostname'
AS d_value
WHERE
name = 'cpu.percent'
AND dimensions ? 'hostname'
FROM
measurements
Dimension Value List Query
(for specific metric and dimension)

Optimisation

CREATE TABLE measurements (
name VARCHAR,
value FLOAT8,
dimensions JSONB,
value_meta JSON
);
CREATE INDEX ON measurements
(name, timestamp);
CREATE INDEX ON measurements USING GIN
(dimensions);
Indexes
Covers all necessary query terms
Using single GIN saves space, but slower

● Series Queries
● All, Group, Specific
● Varying Time Window/Interval
5m|15s, 1h|15s, 1h|300s, 6h|300s, 24h|300s
● Listing Queries
● Metric Names, Dimension Names & Values
● All, Partial

Single
Group
All
0 2000 4000 6000 8000 10000 12000
"Denormalised" Series Queries
5m (15s)
1h (15s)
1h (300s)
6h (300s)
24h (300s)
Duration (ms)

Single
Group
All
0 500 1000 1500 2000 2500
"Denormalised" Series Queries
5m (15s)
1h (15s)
1h (300s)
6h (300s)
24h (300s)
Duration (ms)

Dimension Values
Dimension Names
Metric Names
0 10000 20000 30000 40000 50000 60000
"Denormalised" Listing Queries
All
Partial
Duration (ms)

Dimension Values
Dimension Names
Metric Names
0 1000 2000 3000 4000 5000 6000 7000 8000
"Denormalised" Listing Queries
All
Partial
Duration (ms)

CREATE TABLE measurement_values (
metric_id INT,
value FLOAT8,
value_meta JSON
);
CREATE TABLE metrics (
id SERIAL,
name VARCHAR,
dimensions JSONB,
);
Normalised Schema
Reduces duplication of data
Pre-built set of distinct metric definitions

CREATE FUNCTION get_metric_id (in_name VARCHAR, in_dims JSONB)
RETURNS INT LANGUAGE plpgsql AS $_$
DECLARE
out_id INT;
BEGIN
SELECT id INTO out_id FROM metrics AS m
WHERE m.name = in_name AND m.dimensions = in_dims;
IF NOT FOUND THEN
INSERT INTO metrics ("name", "dimensions") VALUES
(in_name, in_dims) RETURNING id INTO out_id;
END IF;
RETURN out_id;
END; $_$;
Normalised Schema
Function to use at insert time
Finds existing metric_id or allocates new

CREATE VIEW measurements AS
SELECT *
FROM measurement_values
INNER JOIN
metrics ON (metric_id = id);
CREATE INDEX metrics_idx ON
metrics (name, dimensions);
CREATE INDEX measurements_idx ON
measurement_values (metric_id, timestamp);
Normalised Schema
Same queries, use view to join
Extra index to help normalisation step

Single
Group
All
0 500 1000 1500 2000 2500
"Normalised" Series Queries
5m (15s)
1h (15s)
1h (300s)
6h (300s)
24h (300s)
Duration (ms)

Single
Group
All
0 200 400 600 800 1000
"Normalised" Series Queries
5m (15s)
1h (15s)
1h (300s)
6h (300s)
24h (300s)
Duration (ms)

Dimension Values
Dimension Names
Metric Names
0 200 400 600 800 1000
"Normalised" Listing Queries
All
Partial
Duration (ms)

● As time window grows
less detail is necessary, e.g.
● 30s interval at 1 hour
● 300s interval at 6 hour

Timestamp Metric Value
10:00:00 1 10
10:00:00 2 2
10:00:30 1 10
10:00:30 2 4
10:01:30 1 20
10:01:30 2 4
10:02:00 1 15
10:02:00 2 2
10:02:30 1 5
10:02:30 2 2
10:03:00 1 10
10:03:00 2 6
Timestamp Metric Value
10:00:00 1 40
10:00:00 2 10
10:02:00 1 30
10:02:00 2 8

CREATE TABLE summary_values_5m (
metric_id INT,
value_sum FLOAT8,
value_count FLOAT8,
value_min FLOAT8,
value_max FLOAT8,
UNIQUE (metric_id, timestamp)
);
Summarised Schema
Pre-compute every 5m (300s) interval
Functions to be applied must be known

CREATE FUNCTION update_summarise () RETURNS TRIGGER
LANGUAGE plpgsql AS $_$
BEGIN
INSERT INTO summary_values_5m VALUES (
TIME_ROUND(NEW.timestamp, 300), NEW.metric_id,
NEW.value, 1, NEW.value, NEW.value)
ON CONFLICT (metric_id, timestamp)
DO UPDATE SET
value_sum = value_sum + EXCLUDED.value_sum,
value_count = value_count + EXCLUDED.value_count,
value_min = LEAST (value_min, EXCLUDED.value_min),
value_max = GREATEST(value_max, EXCLUDED.value_max);
RETURN NULL;
END; $_$;
Summarised Schema
Entry for each metric/rounded time period
Update existing entries by aggregating

CREATE TRIGGER update_summarise_trigger
AFTER INSERT ON measurement_values
FOR EACH ROW
EXECUTE PROCEDURE update_summarise ();
CREATE VIEW summary_5m AS
SELECT *
FROM
summary_values_5m INNER JOIN metrics
ON (metric_id = id);
Summarised Schema
Trigger applies row to summary table
View mainly for convenience when querying

SELECT
AVG(value) AS avg
FROM
measurements
WHERE
AND '2015-01-01Z06:00:00'
GROUP BY
timestamp
Combined Series Query
Six hour window | Every hostname

SELECT
SUM(value_sum) / SUM(value_count) AS avg
FROM
summary_5m
WHERE
AND '2015-01-01Z06:00:00'
GROUP BY
timestamp
Combined Series Query
Use pre-aggregated summary table
Mostly the same; extra fiddling for AVG

Single
Group
All
0 200 400 600 800 1000
"Summarised" Series Queries
5m (15s)
1h (15s)
1h (300s)
6h (300s)
24h (300s)
Duration (ms)

Dimension Values
Dimension Names
Metric Names
0 200 400 600 800 1000
"Summarised" Listing Queries
All
Partial
Duration (ms)

Summarised
Normalised
Denormalised
0 10000 20000 30000 40000 50000 60000 70000 80000 90000
Ingest Time (1 day / 45M rows)
Seconds

Summarised
Normalised
Denormalised
0 500 1000 1500 2000 2500 3000 3500 4000
Ingest Time (1 day / 45M rows)
Seconds

Summarised
Normalised
Denormalised
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
Disk Usage (1 day / 45M rows)
MB

● Need coarser summaries for wider
queries (e.g. 30m summaries)
● Need to partition data by day to:
● Retain ingest rate due to indexes
● Optimise dropping old data
● Much better ways to produce summaries
to optimise ingest, specifically:
● Process rows in batches of interval size
● Process asynchronous to ingest transaction

Postgres for Log Searching
Requirements
● Central log storage
● Trivially searchable
● Time bounded
● Filter ‘dimensions’
● Interactive query
times (<100ms)

"log": {
"timestamp": 1232141412,
"message":
"Connect from 172.16.8.1:52690 to 172.16.8.10:5000 (keystone/HTTP)",
"dimensions": {
"severity": 6,
"facility": 16,
"pid": "39762",
"program": "haproxy"
"hostname": "dev-controller-0"
},
}
Log Ingest Format
Typically sourced from rsyslog
Varying set of dimensions key/values

CREATE TABLE logs (
message VARCHAR,
dimensions JSONB
);
Basic Schema
Straightforward mapping of source data
Allow for maximum dimension flexibility

connection AND program:haproxy
Query Example
Kibana/Elastic style using PG-FTS
SELECT *
FROM logs
WHERE
TO_TSVECTOR('english', message)
@@ TO_TSQUERY('connection')
AND dimensions @> '{"program":"haproxy"}';

CREATE INDEX ON logs
USING GIN
(TO_TSVECTOR('english', message));
CREATE INDEX ON logs
USING GIN
(dimensions);
Indexes
Enables fast text search on ‘message’
& Fast filtering based on ‘dimensions’

Postgres for Log Parsing
"log": {
"timestamp": 1232141412,
"message":
"dimensions": {
"severity": 6,
"facility": 16,
"pid": "39762",
"program": "haproxy",
"hostname": "dev-controller-0"
},
}

"log": {
"timestamp": 1232141412,
"message":
"dimensions": {
"severity": 6,
"facility": 16,
"pid": "39762",
"hostname": "dev-controller-0",
"tags": [ "connect" ]
},
}

"log": {
"timestamp": 1232141412,
"message":
"dimensions": {
"severity": 6,
"facility": 16,
"pid": "39762",
"hostname": "dev-controller-0",
"tags": [ "connect" ],
"src_ip": "172.16.8.1",
"src_port": "52690",
"dest_ip": "172.16.8.10",
"dest_port": "5000",
"service_name": "keystone",
"protocol": "HTTP"
},
}

….regex!
# SELECT REGEXP_MATCHES(
'Connect from 172.16.8.1:52690 to 172.16.8.10:5000 (keystone/HTTP)',
'Connect from '
|| '(d+.d+.d+.d+):(d+) to (d+.d+.d+.d+):(d+)'
|| ' ((w+)/(w+))'
);
regexp_matches
---------------------------------------------------
{172.16.8.1,52690,172.16.8.10,5000,keystone,HTTP}
(1 row)

Garnish with JSONB
# SELECT JSONB_PRETTY(JSONB_OBJECT(
'{src_ip,src_port,dest_ip,dest_port,service, protocol}',
'{172.16.8.1,52690,172.16.8.10,5000,keystone,HTTP}'
));
jsonb_pretty
-------------------------------
{ +
"src_ip": "172.16.8.1", +
"dest_ip": "172.16.8.10",+
"service": "keystone", +
"protocol": "HTTP", +
"src_port": "52690", +
"dest_port": "5000" +
}
(1 row)

CREATE TABLE logs (
message VARCHAR,
dimensions JSONB
);
Log Schema – Goals:
Parse message against set of patterns
Add extracted information as dimensions

Patterns Table
Store pattern to match and field names
CREATE TABLE patterns (
regex VARCHAR,
field_names VARCHAR[]
);
INSERT INTO patterns (regex, fields_names) VALUES (
'Connect from '
|| '(d+.d+.d+.d+):(d+) to (d+.d+.d+.d+):(d+)'
|| ' ((w+)/(w+))',
'{src_ip,src_port,dest_ip,dest_port,service,protocol}'
);

Log Processing
Apply all configured patterns to new rows
CREATE FUNCTION process_log () RETURNS TRIGGER
LANGUAGE PLPGSQL AS $_$
DECLARE
m JSONB; p RECORD;
BEGIN
FOR p IN SELECT * FROM patterns LOOP
m := JSONB_OBJECT(p.field_names,
REGEXP_MATCHES(NEW.message, p.regex));
IF m IS NOT NULL THEN
NEW.dimensions := NEW.dimensions || m
END IF;
END LOOP;
RETURN NEW;
END; $_$;

CREATE TRIGGER process_log_trigger
BEFORE INSERT ON logs
FOR EACH ROW
EXECUTE PROCEDURE process_log ();
Log Processing Trigger
Apply patterns as messages and extend
dimensions, as inserted into logs table

# INSERT INTO logs (timestamp, message, dimensions) VALUES (
'2017-01-03T06:29:09.043Z',
'Connect from 172.16.8.1:52690 to 172.16.8.10:5000 (keystone/HTTP)',
'{"hostname": "dev-controller-0", "program": "haproxy"}');
# SELECT timestamp, message, JSONB_PRETTY(dimensions) FROM logs;
-[ RECORD 1 ]+------------------------------------------------------------------
timestamp | 2017-01-03 06:29:09.043+00
message | Connect from 172.16.8.1:52690 to 172.16.8.10:5000 (keystone/HTTP)
jsonb_pretty | { +
| "src_ip": "172.16.8.1", +
| "dest_ip": "172.16.8.10", +
| "program": "haproxy", +
| "service": "keystone", +
| "hostname": "dev-controller-0", +
| "protocol": "HTTP", +
| "src_port": "52690", +
| "dest_port": "5000" +
| }

Requirements
● Offload data burden
from producers
● Persist as soon as
possible to avoid loss
● Handle high velocity
burst loads
● Data does not need
to be queryable
Postgres for Queueing

Postgres for Queueing
WITH BINARY
VARCHAR
JSON
JSONB
Denormalised
Normalised
Summarised
0 50 100 150 200 250 300 350 400
Ingest Rate (1d / 45M rows)
K-row/sec

Conclusion… ?
● I view Postgres as a very flexible
“data persistence toolbox”
● ...which happens to use SQL
● Batteries not always included
● That doesn’t mean it’s hard
● Operational advantages of using
general purpose tools can be huge
● Use & deploy what you know & trust

Infrastructure Monitoring with Postgres

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Infrastructure Monitoring with Postgres (20)

Recently uploaded (20)

Infrastructure Monitoring with Postgres