From logs to metrics

From logs to metrics.
@leodido

Who I am.
Leonardo Di Donato
Senior Software Engineer at
Creator of go-syslog
www.linkedin.com/in/leodidonato
twitter.com/leodido
github.com/leodido
@leodido

How many logs
do we generate
every day?
@leodido

The quantity is not the only factor ...
How many standards - if any - we use to log?
How strictly we follow those standards formats?
@leodido

How to transform
kubernetes logs into
metrics with the TICK stack.
Almost everyone needs to govern their logs.
Deriving metrics, synthesizing and visualizing them helps
in decision making.
git.io/k8s-logs-to-metrics-tick (PoC)
@leodido

First of all we needed a log parser.
But to parse which format … ?
BSD-syslog - RFC 3164 - resembled a de-facto standard.
Wide usage, lot of tools, long lifetime span (2001).
But …
messy/informal RFC ...
no strict well-defined grammar
no single stable framing technique
too many customisations around.
Nope!
@leodido

Thus we chose real syslog.
RFC 5424 deprecates RFC 3164
● Well-defined grammar
● Octet counting framing
○ finally the stack trace for a
panic in a single syslog …
● TLS transport mapping
○ secure logs
● Only 9 years old - ie., 2009
@leodido

We chose Ragel to create the (Go) syslog parser
github.com/influxdata/go-syslog
A state machine compiler
● regular languages -> FSM
● can execute code (actions) at arbitrary points
● non-determinism operators
● table or control flow driven state machines
● various host languages - c, c++, obj-c, asm, d, go, java, ruby, c#, ocaml
@leodido

action dgt { printf("DGT: %cn", fc); }
action dec { printf("DEC: .n"); }
action exp { printf("EXP: %cn", fc); }
action exp_sign { printf("SGN: %cn", fc); }
action number { /*NUMBER*/ }
number = (
[0-9]+ $dgt ( '.' @dec [0-9]+ $dgt )?
( [eE] ( [+-] $exp_sign )? [0-9]+ $exp )?
) %number;
main := ( number 'n' )*;
st0:
if ( ++p == pe )
goto out0;
if ( 48 <= (*p) && (*p) <= 57 )
goto tr0;
goto st_err;
tr0:
{ printf("DGT: %cn", (*p)); }
st1:
if ( ++p == pe )
goto out1;
switch ( (*p) ) {
case 10: goto tr5;
case 46: goto tr7;
case 69: goto st4;
case 101: goto st4;
}
if ( 48 <= (*p) && (*p) <= 57 )
goto tr0;
goto st_err;
// ...
The gotos are your best friends. Only when you do not write them!
@leodido

go-syslog provides parsers for RFC 5424 and RFC 5425.
<85>1 2018-10-11T22:14:15.003Z leodido - 31932 - [ex@31932 iut="3"] An auth token...
prival = facility * 8 + severity
● tools.ietf.org/html/rfc5424.html (Syslog grammar)
● tools.ietf.org/html/rfc5425.html (TLS + octet counting)
● tools.ietf.org/html/rfc5426.html (UDP)
● tools.ietf.org/html/rfc6587.html (TCP + octet counting)
● man7.org/linux/man-pages/man3/syslog.3.html
● man7.org/linux/man-pages/man0/syslog.h.0p.html
● man7.org/linux/man-pages/man1/logger.1.html
@leodido

bestEffortOn := true
i := []byte(`<165>1 2018-10-11T22:14:15.003Z mymach.it e - 1 [ex@32473 iut="3"] An app event...`)
p := rfc5424.NewParser()
m, e := p.Parse(i, &bestEffortOn) // best effort mode on means both m and e can have value ...
This results in m being equal to the following SyslogMessage instance. While error e is nil in this case.
// (rfc5424.SyslogMessage)({
// priority: (*uint8)(165),
// facility: (*uint8)(20),
// severity: (*uint8)(5),
// version: (uint16) 1,
// timestamp: (*time.Time)(2018-10-11 22:14:15.003 +0000 UTC),
// hostname: (*string)((len=9) "mymach.it"),
// appname: (*string)((len=1) "e"),
// procID: (*string)(<nil>),
// msgID: (*string)((len=1) "1"),
// structuredData: (*map[string]map[string]string)((len=1) {
// (string) (len=8) "ex@32473": (map[string]string) (len=1) {
// (string) (len=3) "iut": (string) (len=1) "3"
// }
// }),
// message: (*string)((len=33) "An app event...")
// })
@leodido

It provides also a builder.
Incrementally build valid syslog messages
msg := &SyslogMessage{}
msg.SetTimestamp("not a RFC3339MICRO timestamp")
// Not yet a valid message (try msg.Valid())
msg.SetPriority(191)
msg.SetVersion(1)
msg.Valid() // Now it is minimally valid
str, _ := msg.String()
// str is “<191>1 - - - - - -”
Notice that its API ignores input values that does not follow the grammar.
@leodido

Performances.
● ~250ns to parse the smallest legal message
● ~2µs to parse an average legal message
● ~4µs to parse a very long legal message
[no]_empty_input__________________________________-4 30000000 253 ns/op 224 B/op 3 allocs/op
[no]_multiple_syslog_messages_on_multiple_lines___-4 20000000 433 ns/op 304 B/op 12 allocs/op
[no]_impossible_timestamp_________________________-4 10000000 1080 ns/op 528 B/op 11 allocs/op
[no]_malformed_structured_data____________________-4 20000000 552 ns/op 400 B/op 12 allocs/op
[no]_with_duplicated_structured_data_id___________-4 5000000 1246 ns/op 688 B/op 17 allocs/op
[ok]_minimal______________________________________-4 30000000 264 ns/op 247 B/op 9 allocs/op
[ok]_average_message______________________________-4 5000000 1984 ns/op 1536 B/op 26 allocs/op
[ok]_complicated_message__________________________-4 5000000 1644 ns/op 1280 B/op 25 allocs/op
[ok]_very_long_message____________________________-4 2000000 3826 ns/op 2464 B/op 28 allocs/op
[ok]_all_max_length_and_complete__________________-4 3000000 2792 ns/op 1888 B/op 28 allocs/op
[ok]_all_max_length_except_structured_data_and_mes-4 5000000 1830 ns/op 883 B/op 13 allocs/op
[ok]_minimal_with_message_containing_newline______-4 20000000 294 ns/op 250 B/op 10 allocs/op
[ok]_w/o_procid,_w/o_structured_data,_with_message-4 10000000 956 ns/op 364 B/op 11 allocs/op
[ok]_minimal_with_UTF-8_message___________________-4 20000000 586 ns/op 359 B/op 10 allocs/op
[ok]_with_structured_data_id,_w/o_structured_data_-4 10000000 998 ns/op 592 B/op 14 allocs/op
[ok]_with_multiple_structured_data________________-4 5000000 1538 ns/op 1232 B/op 22 allocs/op
[ok]_with_escaped_backslash_within_structured_data-4 5000000 1316 ns/op 920 B/op 20 allocs/op
[ok]_with_UTF-8_structured_data_param_value,_with_-4 5000000 1580 ns/op 1050 B/op 21 allocs/op
@leodido

Telegraf is the plugin-driven server agent for collecting &
reporting metrics.
github.com/influxdata/telegraf
Thus we created the syslog input plugin for it, using go-syslog
● Listens for syslog messages transmitted over UDP - RFC 5426 - or TCP.
● Supports (atm) only messages formatted according to RFC 5424.
● Supports TLS, octet framing (both over TCP - RFC 6587 - and TLS - RFC 5425).
● BSD format - RFC 3164 - in progress.
@leodido

Metrics
Measurement: syslog
● tags
○ severity (string)
○ facility (string)
○ hostname (string)
○ appname (string)
● fields
○ version (integer)
○ severity_code (integer)
○ facility_code (integer)
○ timestamp (integer) - the time recorded in the
syslog message
○ procid (string)
○ msgid (string)
○ sdid (bool)
○ structured data elements (string)
● timestamp - the time the messages was received
@leodido

Input (with octet counting)
169 <165>1 2018-10-01:14:15.000Z mymachine.example.com evntslog - ID47 [exampleSDID@32473
iut="3" eventSource="Application" eventID="1011"] An application event log entry...
Output
syslog,appname=evntslog,facility=local4,hostname=mymachine.example.com,severity=notice
exampleSDID@32473_eventID="1011",exampleSDID@32473_eventSource="Application",exampleSDID@32
473_iut="3",facility_code=20i,message="An application event log
entry...",msgid="ID47",severity_code=5i,timestamp=1065910455003000000i,version=1i
1538421339749472344
@leodido

Our solution
Grab k8s and kernel
logs from journald.
Parse them via telegraf
syslog input plugin.
Visualize logs with
chronograf log viewer.
Elicit new metrics to plot applying a
kapacitor UDF.
Source code at git.io/k8s-logs-to-metrics-tick
@leodido

Using rsyslog to grab RFC 5424 syslog messages from journald.
apiVersion: v1
kind: ConfigMap
metadata:
name: rsyslog
namespace: logging
labels:
component: rsyslog
app: rsyslog
data:
rsyslog.conf: |+
# …
module(load="imjournal" ...)
# This module only works with the journald and json-file docker log drivers
module(load="mmkubernetes" tls.cacert="..." tokenfile="..." annotation_match=["."])
# Extracts k8s metadata
action(type="mmkubernetes")
# …
# Compose RFC5424 message
template(name="rfc5424" type="list") { … }
action(type="omfwd" target="127.0.0.1" port="6514" protocol="tcp" tcp_framing="octet-counted"
template="rfc5424" ...)
@leodido

Setup telegraf syslog plugin to receive log messages over TCP.
apiVersion: v1
kind: ConfigMap
metadata:
name: telegraf
namespace: logging
labels:
component: telegraf
app: telegraf
data:
telegraf.conf: |+
# ...
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
# ...
[[outputs.influxdb]]
urls = ["http://influxdb:8086"] # required
database = "telegraf" # required
retention_policy = "autogen"
write_consistency = "any"
timeout = "1m"
[[inputs.syslog]]
server = "tcp://:6514"
best_effort = true
@leodido

Let’s deploy chronograf and influxDB
apiVersion: v1
kind: Service
metadata:
name: chronograf
namespace: logging
labels:
component: chronograf
app: chronograf
spec:
ports:
- port: 80
targetPort: 8888
name: server
selector:
component: chronograf
---
apiVersion: apps/v1
kind: Deployment
# ...
apiVersion: v1
kind: Service
metadata:
name: influxdb
namespace: logging
labels:
component: influxdb
app: influxdb
annotations:
service.alpha.kubernetes.io/tolerate-unready-endpoints: “true”
spec:
clusterIP: None
ports:
- port: 8086
name: server
selector:
component: influxdb
---
apiVersion: apps/v1
kind: StatefulSet
# ...
@leodido

Logs are just a mess!
Inspecting logs coming from a single server is easy.
Inspecting logs coming from a distributed system is hard.
We need metrics!
@leodido

Now we want to detect and count the OOMs.
Logs are streams.
We need a streaming processor!
github.com/influxdata/kapacitor
A streaming processor can be programmed to identify
the patterns we want and act on them, e.g: OOM Kills.
Memory cgroup out of memory: Kill process 13012 (stress) score 1958 or sacrifice child
@leodido

Let’s write a tick script to grab log points
dbrp "telegraf"."autogen"
stream
|from()
.measurement('syslog')
.truncate(1ms)
.where(lambda: "appname" == 'kernel') # filter by points tag
.where(lambda: "message" =~ /sacrifice/) # filtering on messages with regex
@example() # passing points to our user defined function (UDF)
|influxDBOut()
.database('telegraf')
.measurement('k8s')
Collection Ingestion Storage
Stream
reprocessing
Dashboards
@leodido

Let’s configure kapacitor
# …
[udf]
[udf.functions]
[udf.functions.example]
socket = "/tmp/example.sock"
timeout = "10s"
[[influxdb]]
enabled = true
default = true
name = "logging"
urls = ["http://localhost:8086"]
timeout = 0
startup-timeout = "5m"
[influxdb.subscriptions]
telegraf = ["autogen"]
@leodido

Let’s write the UDF
func (h *handler) Point(p *agent.Point) error {
var r = regexp.MustCompile(`(?m).*Kill process (?P<pid>d+) (?P<proc>(.*)) score (?P<score>d+)`)
message, ok := p.FieldsString[“message”]
if ok {
m := r.FindStringSubmatch(message)
data := mapSubexpNames(m, r.SubexpNames())
proc := strings.Trim(data[“proc”], “()”)
state := h.state[proc]
if state == nil {
state := &myState{Counter: 0}
h.state[proc] = state
}
h.state[proc].update()
newpoint := &agent.Point{
Time: time.Now().UnixNano(),
Tags: map[string]string{
“proc”: proc,
“pid”: string(data[“pid”]),
},
FieldsInt: map[string]int64{
“count”: h.state[proc].Counter,
},
}
// Send point
h.agent.Responses <- &agent.Response{
Message: &agent.Response_Point{
Point: newpoint,
},
}
}
return nil
}
@leodido

Thanks.
@leodido
git.io/k8s-logs-to-metrics-tick
git.io/go-syslog
github.com/influxdata

From logs to metrics

More Related Content

What's hot (9)

Similar to From logs to metrics (20)

More from Leonardo Di Donato (9)

Recently uploaded (20)

From logs to metrics