rsyslog v8: more than just syslog!

rsyslog v8: 
more than just syslog!
Yury Bushmelev

Lazada

what people usually think
rsyslog is
/dev/log
udp:514
some

magic
/var/log/some.log
remote udp:514

"some magic"
queue
input
pre-
processor
action
queue
parser 
&
ﬁlter
action
processor
output
input
input
action
queue
action
queue
action
processor
action
processor
output
output
ruleset

rulesets
input(...)
input(...)
if ($syslogtag ==
"prog1") then {
action(...)
} else {
action(...)
}
input(...
ruleset="rules1")
input(...
ruleset="rules2")
ruleset(name="rules1") {
action(...)
}
ruleset(name="rules2") {
action(...)
}

configuration formats
• basic AKA sysklogd (*.* /var/log/file.log) – still OK in
simple cases

• obsolete legacy ($UglyScaryThings everywhere) – Do not
use it at all! "It will make your life miserable" © Rainer
Gerhards

• advanced AKA RainerScript – new (2010) readable
configuration language

queues
• most confusing part of rsyslog conﬁguration

queues

• 2 places

• ruleset queue

• action queue

queues

• 2 places

• ruleset queue

• action queue

• 4 modes

• direct (AKA no queue, sync)

• in-memory

• on-disk

• disk-assisted (DA, in-memory + on-disk)

queues and performance
• try to increase ruleset queue workers number first

• action queue is mostly about string building

• for fast action (omfile) thread syncing overhead may be
significant

• for single-action ruleset everything is complicated...

• read more details in Rainer's blog: http://
blog.gerhards.net/2013/06/rsyslog-performance-main-
and-action.html

– ELK?
• read log messages from ﬁle/network

• parse

• modify

• convert to JSON

• send to ElasticSearch

– ERK!
• read log messages from ﬁle/network (inputs)

• parse (builtin parser/RainerScript/mm* modules)

• modify (RainerScript/mm* modules/templates)

• convert to JSON (templates)

• send to ElasticSearch (omelasticsearch)

parse (mmnormalize)
• "mmnormalize" is unique feature of rsyslog based on
liblognorm library which...

• ... is amazingly fast

• ... because it's using parse tree (not regexp/grok)

• ... may try multiple rules for same message (ﬁrst win)

• ... have some documentation: 
http://www.liblognorm.com/ﬁles/manual/index.html

liblognorm rules
version=2
rule=v0:MyAuthAPI: Invalid user %user:word%
from %src-ip:ipv4%
rule=v1:MyAuthAPI[%pid:number%]: Invalid
user %user:word% from %src-ip:ipv4%

modify/send as JSON v1
template(name="tmpl_to_json_v1" type="string" string="%$!all-json%")
template(name="timereported_rfc3339" type="string"
string="%timereported:::date-rfc3339%")
ruleset(name="test") {
action(type="mmjsonparse")
set $!_log_time = exec_template("timereported_rfc3339");
set $!_log_tag = $!tag;
unset $!tag;
...
action(type="omelasticsearch"
server="es.example.com"
template="tmpl_to_json_v1"
...)
}

modify/send as JSON v2
template(name="tmpl_to_json_v2" type="list" option.json="on") {
constant(value="{")
property(outname="_log_time" name="timereported" dateFormat="rfc3339"
format="jsonf")
constant(value=",")
property(outname="_log_tag" name="tag" format="jsonf")
constant(value=",")
property(outname="message" name="msg" dropLastLF="on" format="jsonf")
constant(value="}")
}
ruleset(name="test") {
action(type="omelasticsearch"
server="es.example.com"
template="tmpl_to_json_v2"
...)
}

reliable delivery
• message will be lost when using UDP delivery

• message may be lost when using TCP delivery

reliable delivery


• ... so there is RELP (imrelp/omrelp)

• ... it may do TLS with compression easy as well

reliable delivery


• ... so there is RELP (imrelp/omrelp)

• ... it may do TLS with compression easy as well

• ... but it's single-threaded :(

metrics (impstats)
{ "name": "global", "origin": "dynstats", "values": { } }
{ "name": "imuxsock", "origin": "imuxsock", "submitted": 3612711,
"ratelimit.discarded": 0, "ratelimit.numratelimiters": 0 }
{ "name": "action 0", "origin": "core.action", "processed":
33429, "failed": 0, "suspended": 0, "suspended.duration": 0,
"resumed": 0 }
{ "name": "resource-usage", "origin": "impstats", "utime":
290424444, "stime": 314385112, "maxrss": 3176, "minflt": 509,
"majflt": 105, "inblock": 5920, "oublock": 11030360, "nvcsw":
7281385, "nivcsw": 7506 }
{ "name": "main Q", "origin": "core.queue", "size": 0,
"enqueued": 3612765, "full": 0, "discarded.full": 0,
"discarded.nf": 0, "maxqsize": 30 }
• read more here: https://www.rsyslog.com/doc/v8-stable/conﬁguration/
rsyslog_statistic_counter.html

metrics (dyn_stats)
dyn_stats(name="msg_per_host" resettable="on"
maxCardinality="3000" unusedMetricLife="600")
set $.inc = dyn_inc("msg_per_host", $hostname);
# impstats output
# { "name": "msg_per_host", "origin":
"dynstats.bucket", "values": { "lab3.sgdc":
1002540, "lab4.sgdc": 1318551, "lab7.sgdc":
1788136} }

lookup tables
{ "version" : 1,
"nomatch" : "unknown",
"type" : "string",
"table" : [
{"index" : "10.0.1.1", "value" : "A" },
{"index" : "10.0.1.2", "value" : "A" },
{"index" : "10.0.1.3", "value" : "A" },
{"index" : "10.0.2.1", "value" : "B" },
{"index" : "10.0.2.2", "value" : "B" },
{"index" : "10.0.2.3", "value" : "B" }]}

lookup tables
lookup_table(name="host_bu"
file="/var/lib/host_billing_unit_mapping.json"
reloadOnHUP="on"
)
set $.bu = lookup("host_bu", $hostname);
if ($.bu != "unknown") then {
...
}
# Go to "bu_A"/"bu_B"/"bu_unknown" ruleset:
# call_indirect "bu_" & $.bu;

lookup tables
if ($.do_reload == "y") then {
reload_lookup_table("host_bu", "unknown")
}

runtime control
input(type="imuxsock" socket="/run/rsyslog-control.sock"
ruleset="control")
ruleset(name="control") {
if ($msg == "reload bu") then {
reload_lookup_table("host_bu", "unknown")
} else if ($msg == "run cmd") then {
action(type="omprog" binary="/path/to/script.sh")
} else {
action(type="omfile" file="/var/log/rsyslog-control.log")
}
}

any questions?
• Yury Bushmelev

• https://twitter.com/Jay7t

• https://github.com/jay7x

rsyslog v8: more than just syslog!

More Related Content

What's hot (20)

Similar to rsyslog v8: more than just syslog! (20)

Recently uploaded (20)

rsyslog v8: more than just syslog!