Best Practices for Forwarder Hierarchies

© 2 0 2 0 S P L U N K I N C .
© 2 0 2 0 S P L U N K I N C .
Best Practices
for Forwarder
Hierarchies
Richard Morgan
Principal Architect | Splunk

During the course of this presentation, we may make forward‐looking statements regarding
future events or plans of the company. We caution you that such statements reflect our
current expectations and estimates based on factors currently known to us and that actual
events or results may differ materially. The forward-looking statements made in the this
presentation are being made as of the time and date of its live presentation. If reviewed after
its live presentation, it may not contain current or accurate information. We do not assume
any obligation to update any forward‐looking statements made herein.
In addition, any information about our roadmap outlines our general product direction and is
subject to change at any time without notice. It is for informational purposes only, and shall
not be incorporated into any contract or other commitment. Splunk undertakes no obligation
either to develop the features or functionalities described or to include any such feature or
functionality in a future release.
Splunk, Splunk>, Data-to-Everything, D2E, and Turn Data Into Doing are trademarks and registered trademarks of Splunk Inc. in the United States
and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2020 Splunk Inc. All rights reserved.
Forward-
Looking
Statements
© 2 0 2 0 S P L U N K I N C .

© 2 0 2 0 S P L U N K I N C .
Principal Architect | Splunk
Richard Morgan

© 2 0 2 0 S P L U N K I N C .
Spreading That Splunk Across EMEA Since 2013
Self professed data junkie and SPL addict
CEODog trainer Father

© 2 0 2 0 S P L U N K I N C .
indexers
forwarders
Search heads (you are here!)
A Splunk
installation is
much like an
iceberg, the visible
tip is the indexers,
search heads,
cluster master etc.

© 2 0 2 0 S P L U N K I N C .
Why is Event
Collection
Tuning
Important?
• Data collection is the foundation
of any Splunk instance
• Event distribution underpins
linear scaling of indexing and
search
• Events must be synchronized in
time to corelate across hosts
• Events must arrive in a timely
fashion for alerts to be effective

© 2 0 2 0 S P L U N K I N C .
What is
Event Distribution?

© 2 0 2 0 S P L U N K I N C .
What is Good Event Distribution?
Event distribution is how Splunk spreads its incoming data across multiple indexers
IndexerIndexerIndexerIndexer
Forwarder
25 MB 25 MB 25 MB
100 MB
25 MB

© 2 0 2 0 S P L U N K I N C .
Why is Good Event Distribution Important?
Search
head
Event distribution is critical for the even distribution of search (computation) workload
10 sec10 sec10 sec10 sec
11 sec
Execution of a search for
a fix time range

© 2 0 2 0 S P L U N K I N C .
What is ‘Bad’ Event Distribution?
Bad event distribution is when the spread of events is uneven across the indexers
Forwarder
10 MB 10 MB 10 MB
100 MB
70 MB

© 2 0 2 0 S P L U N K I N C .
Bad Event Distribution Affects Search
Search
head
Search time becomes unbalanced, searches take longer to complete and reducing throughput
10 sec10 sec10 sec90 sec
91 sec
Execution of a search for
a fix time range

© 2 0 2 0 S P L U N K I N C .
What is Event Delay?

© 2 0 2 0 S P L U N K I N C .
What is Event Delay?
IUF Indexer
Search
head
UFApp Log file readwrite s2s s2s search
The buffer is
appended to the
log file and the
operating
system notifies
the UF of the
change.
The delta is
read, split up
into 64kb blocks,
compressed and
transmitted with
some additional
meta data
Decodes S2S,
processes,
routes, encodes
S2S, retransmits
streams of data
Decodes S2S
parses events
from streams,
via line breakers,
and time
extraction.
Creates indexes
and writes to
disk
The app needs to
write out log events,
it creates a buffer
and then flushes it
Sends search
request to
indexers, which
open buckets
files and and
gets results
returns them SH

© 2 0 2 0 S P L U N K I N C .
What is Good Event Delay?
It should take between 3-5 seconds from event generation to that
event being searchable
3-5 seconds
IUF Indexer
Search
head

© 2 0 2 0 S P L U N K I N C .
How Do You Calculate Event Delay?
| eval event_delay=_indextime - _time
The timestamp of
the event is _time
The time of
indexing is
_indextime
The time
of search
is now()
IUF Indexer
Search
head

© 2 0 2 0 S P L U N K I N C .
How Streams Affect
Event Distribution

© 2 0 2 0 S P L U N K I N C .
autoLBFrequency (outputs.conf)
autoLBFrequency = <integer>
* The amount of time, in seconds, that a forwarder sends data to an indexer
before redirecting outputs to another indexer in the pool.
* Use this setting when you are using automatic load balancing of outputs
from universal forwarders (UFs).
* Every 'autoLBFrequency' seconds, a new indexer is selected randomly from the
list of indexers provided in the server setting of the target group
stanza.
* Default: 30
30 seconds of 1 MB/s is 30 MB for each connection!

© 2 0 2 0 S P L U N K I N C .
Data is Distributed Across
the Indexers Over Time
Each indexer is allocated 30s of data via Randomized Round Robin
Indexer 1 Indexer 2 Indexer 3 Indexer 4
0-30 sec 30-60 sec 60-90 sec 90-180 sec
0 sec 30 sec 60 sec 90 sec 120 sec 180 sec 270 sec 330 sec

© 2 0 2 0 S P L U N K I N C .
earliest=now latest=-90s
Indexer 4 has no data to process
0-30 sec 30-60 sec 60-90 sec 90-180 sec

© 2 0 2 0 S P L U N K I N C .
earliest=now latest=-180s
Indexer 4 has 2x the data to process
0-30 sec 30-60 sec 60-90 sec 90-180 sec

© 2 0 2 0 S P L U N K I N C .
Event Distribution Improves Over Time
Switching at 30 seconds
5 mins = 10 connections
1 hour = 120 connections
1 day = 2880 connections
How long before all indexers have data?
10 indexers = 5 mins
Larger clusters take longer to get “good” event distribution
UF
idx
idx
idx
30 sec 30 sec

© 2 0 2 0 S P L U N K I N C .
Event Distribution Improves Over Time
time
Standarddeviation
Initially we start connecting to just one
indexer and the standard deviation is high
As time goes to infinity
standard deviation
should trend to zero
Most searches execute
over the last 15mins
Standarddeviationacrossindexers
As time passes the standard
deviation improves

© 2 0 2 0 S P L U N K I N C .
It’s Not Round Robin (RR), it’s Randomized RR
Round robin:
Round 1 order: 1,2,3,4
Randomized Round Robin:
Splunk’s randomized round robin algorithm quickens event distribution

© 2 0 2 0 S P L U N K I N C .
Types of Data Flows

© 2 0 2 0 S P L U N K I N C .
Types of Data Flow Coming into Splunk
Periodic data, typically a scripted
input, very spikey. Think AWS
CloudWatch, think event delay.
Constant data rates, nice and
smooth, think metrics.
Variable data rates, typically
driven by usage, think web logs
One shot, much like periodic data
It can be difficult to optimize for every type of data flow
Time
Datarate
One shot/bulk load Periodic data
Constant data rates
Variable data rates

© 2 0 2 0 S P L U N K I N C .
Time Based LB Does Not Work Well on Its Own
timebasedAutoLB is the default and is set to 30s
Constant data
Variable data
rates Periodic data
One shot/bulk
load

© 2 0 2 0 S P L U N K I N C .
autoLBVolume (outputs.conf)
autoLBVolume = <integer>
* The volume of data, in bytes, to send to an indexer before a new indexer
is randomly selected from the list of indexers provided in the server
setting of the target group stanza.
* This setting is closely related to the 'autoLBFrequency' setting.
The forwarder first uses 'autoLBVolume' to determine if it needs to switch to another
indexer. If the 'autoLBVolume' is not reached,
but the 'autoLBFrequency' is, the forwarder switches to another indexer as the
forwarding target.
* A non-zero value means that volume-based forwarding is active.
* 0 means the volume-based forwarding is not active.
* Default: 0
Switching too fast can result in lower throughput
Best practice !!!

© 2 0 2 0 S P L U N K I N C .
autoLBVolume + time based is Much Better
autoLBVolume is better for variable and bursty data flows
Constant data
Variable data
rates Periodic data
One shot/bulk
load

© 2 0 2 0 S P L U N K I N C .
How to Measure
Event Distribution

© 2 0 2 0 S P L U N K I N C .
Call the REST API to Get RT Ingestion
Excellent
| rest /services/server/introspection/indexer
| eventstats stdev(average_KBps) avg(average_KBps)

© 2 0 2 0 S P L U N K I N C .
Ingestion Over Time
index=_internal Metrics TERM(group=thruput) TERM(name=thruput)
sourcetype=splunkd
[| dbinspect index=*
| stats values(splunk_server) as indexer
| eval host_count=mvcount(indexer),
search="host IN (".mvjoin(mvfilter(indexer!=""), ", ").")"]
| eval host_pipeline=host."-".ingest_pipe
| timechart minspan=30sec limit=0 per_second(kb) by host_pipeline
Mostly good but some horrid spikes
Too much data
being ingested by a
single indexer

© 2 0 2 0 S P L U N K I N C .
Indexer Thruput Search Explained
index=_internal Metrics TERM(group=thruput) TERM(name=thruput)
sourcetype=splunkd
| eval host_pipeline=host."-".ingest_pipe
| timechart minspan=30sec limit=0 per_second(kb) by host_pipeline
Use TERM to speed up search
execution
Use metrics to get pipeline
statistics
Sub-search returns indexer listMeasures each pipeline individually

© 2 0 2 0 S P L U N K I N C .
Count the Events per Indexer via Job Inspector
For any search, open the job inspector and
see how many events each indexer returned.
This should be approximately the same for
each indexer.
In a multi site or multi cluster environment we
expect that groups of indexers are likely to
have different numbers of events.
Within a single site the number of events
should be the about same for each indexer.

© 2 0 2 0 S P L U N K I N C .
Count the events per indexer and index via searchCount the Events per Indexer and Index
via Search
The event distribution
score per index

© 2 0 2 0 S P L U N K I N C .
Calculate Event Distribution per Index
for -5mins
| tstats count where
index=*
splunk_server=*
host=* earliest=-5min latest=now
by splunk_server index
| stats
sum(count) as total_events
stdev(count) as stdev
avg(count) as average
by index
| eval normalized_stdev=stdev/average
Event distribution can be measured per index and per host
Narrow indexes
Narrow to site or cluster
Calculate score
Calculate per index, or host

© 2 0 2 0 S P L U N K I N C .
Shameless Self Promotion
Use my event distribution measurement dashboard to assess your stack
http://bit.ly/2WxRXvI

© 2 0 2 0 S P L U N K I N C .
How to Configure the Event Distribution
Dashboard
4. Set the rate of increase in durations as a power function
1. Select the site or cluster to
analyze
6. Click link to run analysis
5. Click on the number of steps
to generate
2. Select the indexes present in
the selected site or cluster
3. Set starting duration, one
second is the default

© 2 0 2 0 S P L U N K I N C .
How to Read the First Panel
Variation of events across the indexers
Data received an indexer in -1 sec
Logscaleofeventsreceived
stdev improves as time increases
Each series is a time
range, exponentially
growing.
Each column is an indexer
Data received an indexer in -54 mins

© 2 0 2 0 S P L U N K I N C .
How to Read the Second Panel
Events scanned in each step
Each series is an index
Thenumberofevents
received(logscale)
After ~8mins indexer received
100k events into index
Each time series plotted on x axis

© 2 0 2 0 S P L U N K I N C .
How to Read the Third Panel
How many indexers received data in time range
All 9 indexers received data into
every index within one second

© 2 0 2 0 S P L U N K I N C .
How to Read the Final Panel
How event distribution is improving over time
3% variation after 5 mins
10% variation after 5 mins > 1% variation after 54 mins

© 2 0 2 0 S P L U N K I N C .
Three hours before all
indexers have data,
need to accelerate to
improve entropy
Randomization is not
happening fast
enough
Incoming data rates for
indexers very different for the
different indexes

© 2 0 2 0 S P L U N K I N C .
Randomization is slow for
most indexes and
plateaus for some
Some indexers are not
getting data, likely
misconfigured
forwarders

© 2 0 2 0 S P L U N K I N C .
One index is not
improving fast
enough
Near Perfect Event Distribution Using
Intermediate Forwarders

© 2 0 2 0 S P L U N K I N C .
Measuring Event Delay

© 2 0 2 0 S P L U N K I N C .
Use TSTATS to Compute Event Delay at Scale

© 2 0 2 0 S P L U N K I N C .
Delay per index and host
Use tstats because indextime as an indexed field!
| tstats max(_indextime) as indexed_time count
where index=*
latest=+1day earliest=-1day
_index_latest=-1sec _index_earliest=-2sec
by index host splunk_server _time span=1s
| eval _time=round(_time), delay=indexed_time-
_time,
delay_str=tostring(delay,"duration")
| eventstats max(delay) as max_delay
max(_time) as max_time
count as eps
by host index
| where max_delay = delay
| eval max_time=_time
| sort - delay

© 2 0 2 0 S P L U N K I N C .
Delay per index and host
| tstats max(_indextime) as indexed_time count where index=*
latest=+1day earliest=-1day _index_latest=-1sec _index_earliest=-2sec
by index host splunk_server _time span=1s
| eval _time=round(_time), delay=indexed_time-_time,
delay_str=tostring(delay,"duration")
| eventstats max(delay) as max_delay
max(_time) as max_time
count as eps
by host index
| where max_delay = delay
| eval max_time=_time
| sort - delay
Get events received in the last
second, irrespective of the delay
max(_indextime) is
latest event
Split by _time to a
resolution of a second
Drop subseconds
Compute the delta

© 2 0 2 0 S P L U N K I N C .
More Shameless Self Promotion
I have a dashboards that use this method to measure delay
http://bit.ly/2R9i4Yx
http://bit.ly/2I8qtIS

© 2 0 2 0 S P L U N K I N C .
How to Use the Event Delay Dashboard
Drills down to per host delay
Select indexes to measure
Keep time range short,
but extend to capture
periodic data
translates into
_indextime, not _time

© 2 0 2 0 S P L U N K I N C .
How to Read the Main Panels
By default the results are all for the last second
The average
delay per index
The number of
events received
per index
The maximum
delay per index
The number of
hosts per index
How events are distributed across the cluster

© 2 0 2 0 S P L U N K I N C .
Select an Index by Clicking on a Series
Click on any chart to select an index
Click on a host to drill down
Select period for drill down
Sorted by descending delay

© 2 0 2 0 S P L U N K I N C .
Drill Down to Host to Understand Reasons
for Delay
The number of
events generated
at time, per index
The number of
events
generated at
time, by delay
The number of
events received
at time, per index
The number of
events
received at
time, by delay

© 2 0 2 0 S P L U N K I N C .
The Causes of
Event Delay

© 2 0 2 0 S P L U N K I N C .
Congestion
network latency, IO contention, pipeline congestion, CPU saturation, rate limiting
Clock skew
Time zones wrong, clock drift, parsing issue
Timeliness
Histortical load, polling APIs, scripted inputs, component restarts

© 2 0 2 0 S P L U N K I N C .
Congestion: Rate Limiting
Rate limiting slows transmission and true event delay occurs
Default maxkbps=256
IUF Indexer
Search
head

© 2 0 2 0 S P L U N K I N C .
Congestion: Network
Network saturation acts like rate limiting and causes true event delay
Network congestion
IUF Indexer
Search
head

© 2 0 2 0 S P L U N K I N C .
Congestion: Indexer
Excessive ingestion rates, FS IO problems, inefficient regex, inefficient
line breaking cause all cause true event delay
Pipeline
congestion
IUF Indexer
Search
head

© 2 0 2 0 S P L U N K I N C .
Timeliness: Scripted Inputs
Increase polling frequency to reduce event delay
Polling very x mins means delays
of up to x mins for each
IUF Indexer
Search
head
UFApp Periodic API read s2s s2s search

© 2 0 2 0 S P L U N K I N C .
Timeliness: Offline Components
Restarting forwarders causes brief event delay
Forwarder must be
running or events queue
IUF Indexer
Search
head

© 2 0 2 0 S P L U N K I N C .
Timeliness: Historical Data
Loading historical data creates fake event delay
Backfilling data
from the past
IUF Indexer
Search
head

© 2 0 2 0 S P L U N K I N C .
Clock Skew: Time Zones
When time zones aren’t configured correctly event delay measurement
is shifted into the past or future
Time zone A
Time zone B
IUF Indexer
Search
head

© 2 0 2 0 S P L U N K I N C .
Clock Skew: Drift
Use NTP to align all clocks across your estate to maximize the
usefulness of Splunk
8:00 pm
7:59 pm
When events appear to arrive slightly from
the future or past, this makes time-based
coloration hard
IUF Indexer
Search
head

© 2 0 2 0 S P L U N K I N C .
Clock Skew: Date Time Parsing Problems
Automatic source typing assumes American date format when ambiguous
Always explicitly set the date time
exactly per sourcetype
IUF Indexer
Search
head

© 2 0 2 0 S P L U N K I N C .
Reasons for Poor
Event Distribution

© 2 0 2 0 S P L U N K I N C .
Sticky forwarders
Super giant forwarders
Badly configured intermediate forwarders
Indexer abandonment
Indexer starvation
Network connectivity problems
Single target
Forwarder bugs
TCPback off
maxKBps
Channel saturation
HEC ingestion

© 2 0 2 0 S P L U N K I N C .
Sticky Forwarders

© 2 0 2 0 S P L U N K I N C .
The UF Uses “Natural Breaks” to
Chunk Up Logs
10 events 50kb
20 events 80kb
10k events
10MB
10 events 50kb
My logging
application
Logging Semantics:
1. Open file
2. Append to file
3. Close file
We know we have a
natural break each time
the file is closed.
Target
1
Time
Target
2
Target
4
Target
3
Natural break
Natural break
Natural break
EOF
When an application
bursts the forwarder is
forced to “stick” to the
target until the
application generates a
natural break

© 2 0 2 0 S P L U N K I N C .
The Problem is Exasperated with IUFs
UF 1
Indexer
1IUF 1
HWF 1
IUF 2
UF 3
Indexer
2
An intermediate universal
forwarder (IUF) works with
unparsed streams and
can only switch away the
incoming stream contains a
break.
Connections can last for
hours and this causes bad
event distribution
UF 2
IUFs cannot afford to be sticky and must switch on time
stuck

© 2 0 2 0 S P L U N K I N C .
The Problem Doesn’t Exist with an
Intermediate HWF
UF 1
Indexer
1HWF 1
HWF 1
HWF 2
UF 3
Indexer
2
The HWF parses data and
forwarder events, not
streams.
HWF can receive
connections from sticky
forwarders causing
throughput problems
UF 2
Heavy forwarders are generally considered a bad practise

© 2 0 2 0 S P L U N K I N C .
forceTimebasedAutoLB (outputs.conf)
forceTimebasedAutoLB = <boolean>
* Forces existing data streams to switch to a newly elected indexer every
auto load balancing cycle.
* On universal forwarders, use the 'EVENT_BREAKER_ENABLE' and
'EVENT_BREAKER' settings in props.conf rather than
'forceTimebasedAutoLB'
for improved load balancing, line breaking, and distribution of events.
* Default: false¯¯
Forcing a UF to switch can create broken events and generate parsing errorsz

© 2 0 2 0 S P L U N K I N C .
How forceTimeBasedAutoLB Works
Splunk to Splunk protocol (s2s) uses datagrams of 64KB
64KB 64KB 64KB 64KB 64KB
Log
sources
current
indexer
next
indexer
64KB
First event Remaining events
30sec
Provided that an events doesn’t succeed a s2s datagram the algorithm works perfectly
We need to switch, but
there is no natural
break!
Send packet to both
assuming that the packet
contains multiple events
and a line break will be
found
Event
boundary
Reads
up to 1st
event
Ignores up
to 1st event

© 2 0 2 0 S P L U N K I N C .
Applying forceTimeBasedAutoLB to IUF
UF 1
Indexer
1IUF 1
HWF 1
IUF 2
UF 3
Indexer
2
The Intermediate
Universal Forwarder
will force switching
without a natural
break.
The indexers no
longer get sticky
sessions!
UF 2
The intermediate forwarders still receive sticky sessions

© 2 0 2 0 S P L U N K I N C .
EVENT_BREAKER (outputs.conf)
# Use the following settings to handle better load balancing from UF.
# Please note the EVENT_BREAKER properties are applicable for Splunk Universal
# Forwarder instances only.
EVENT_BREAKER_ENABLE = [true|false]
* When set to true, Splunk software will split incoming data with a
light-weight chunked line breaking processor so that data is distributed
fairly evenly amongst multiple indexers. Use this setting on the UF to
indicate that data should be split on event boundaries across indexers
especially for large files.
* Defaults to false
# Use the following to define event boundaries for multi-line events
# For single-line events, the default settings should suffice
EVENT_BREAKER = <regular expression>
* When set, Splunk software will use the setting to define an event boundary at
the end of the first matching group instance.
EVENT_BREAKER is configured with a regex per sourcetype

© 2 0 2 0 S P L U N K I N C .
EVENT_BREAKER is Complicated to
Maintain on IUF
UF1 – 5 source types
IUF1
65 source
types
Configure each
sourcetype with
EVENT_BREAKER so
the forwarder doesn’t
get stuck the IUF.
Trying to maintain
EVENT_BREAKER on an
IUF can be impractical due
to aggregation of streams
and source types.
If the endpoints all
implement
EVENT_BREAKER, it
won’t be triggered on the
IUF
forceTimeBasedAutoLB is universal algorithm, EVENT_BREAKER is not

© 2 0 2 0 S P L U N K I N C .
The Final Solution to Sticky Forwarders
UF 1
Indexer
1IUF 1
HWF 1
IUF 2
UF 3
Indexer
2
Removal of sticky
forwarders will lower
event delay, improve
event distribution, and
improve search
execution times
UF 2
Configured correctly intermediate forwarders are great for improving event distribution.
Configure EVENT_BREAKER per
sourcetype, use autoLBVolume
Increase autoLBFrequency,
enable forceTimeBasedAutoLB, use
autoLBVolume

© 2 0 2 0 S P L U N K I N C .
Find All Sticky Forwarders by Host Name
index=_internal sourcetype=splunkd
TERM(eventType=connect_done) OR
TERM(eventType=connect_close)
| transaction
startswith=eventType=connect_done
endswith=eventType=connect_close
sourceHost sourcePort host
| stats stdev(duration) median(duration) avg(duration) max(duration)
by sourceHost
| sort - max(duration)
Intermediate LB will invalidate results

© 2 0 2 0 S P L U N K I N C .
Find All Sticky Forwarders by Host Name
index=_internal sourcetype=splunkd
(TERM(eventType=connect_done) OR
TERM(eventType=connect_close) OR
TERM(group=tcpin_connections))
| transaction
startswith=eventType=connect_done
endswith=eventType=connect_close
sourceHost sourcePort host
| stats stdev(duration) median(duration) avg(duration) max(duration)
by hostname
| sort - max(duration)
Error prone as it requires that hostname = host

© 2 0 2 0 S P L U N K I N C .
Not All Forwarders are Born Equal
AWS
Unix
Hosts
Network
Syslog +
UF
HWF
Data
bases
Splunk
Cluster
Windows
host
5 KB/s
Windows
host
Windows
host
Super giant forwarder make others look like rounding errors

© 2 0 2 0 S P L U N K I N C .
Understanding Forwarder Weight DistributionFwd1
Fwd2
Fwd3
Fwd
4
Fwd
5
Reading from left to right
• 0% forwarders = 0% data
• 100% forwarders 100% data
We can plot these pairs as an chart to
normalize and compare stacks.
10 MB/s
3 MB/s
1 MB/s
500 KB/s 500 KB/s
Total ingestion is 15 MB/s
0% 100%50%

© 2 0 2 0 S P L U N K I N C .
Plotting Normalized Weight Distribution
0% 100%
0%
66%
90%
100%
Distance from
“perfect”
When all forwarders share
the same data rates we
get a straight line

© 2 0 2 0 S P L U N K I N C .
Examples of Forward Weight Distribution
Data imbalance issues can be found on very large stack and must be addressed
10% 100%

© 2 0 2 0 S P L U N K I N C .
Forwarder Weight Distribution Search
index=_internal Metrics sourcetype=splunkd TERM(group=tcpin_connections) earliest=-4hr latest=now
[| dbinspect index=_*
| eval search="host IN (".mvjoin(mvfilter(indexer!=""), ", ").")"]
| stats sum(kb) as throughput
by hostname
| sort - throughput
| eventstats sum(throughput) as total_throughput
dc(hostname) as all_forwarders
| streamstats sum(throughput) as accumlated_throughput count
by all_forwarders
| eval coverage=accumlated_throughput/total_throughput,
progress_through_forwarders=count/all_forwarders
| bin progress_through_forwarders bins=100
| stats max(coverage) as coverage
by progress_through_forwarders all_forwarders
| fields progress_through_forwarders coverage

© 2 0 2 0 S P L U N K I N C .
Find Super Giant Forwarders and Reconfigure
1. Configure EVENT_BREAKER and / or forceTimeBasedAutoLB
2. Configure multiple pipelines (validate that they are being used)
3. Configure autoLBVolume and / or increase switching speed (keeping an eye on
throughput)
4. Use INGEST_EVAL and random() to shard output data flows
Super giant forwarders need careful configuration

© 2 0 2 0 S P L U N K I N C .
Firewalls Can Block Connections
Indexer A Indexer B Indexer C
Forwarder
Forwarders generate errors when they cannot connect
A common cause for starvation is forwarders only
able to connect to a subset of indexers due to
network problems.
Normally a firewall or LB is blocking the
connections
This is very common when the indexer cluster is
increased in size.
replication
Network
says ”no”
Outputs:
A+B+C

© 2 0 2 0 S P L U N K I N C .
Find Forwarders Suffering Network Problems
index=_internal earliest=-24hrs latest=now sourcetype=splunkd
TERM(statusee=TcpOutputProcessor) TERM(eventType=*)
| stats count
count(eval(eventType="connect_try")) as try
count(eval(eventType="connect_fail")) as fail
count(eval(eventType="connect_done")) as done
by destHost destIp
| eval bad_output=if(try=failed,"yes","no")
Search for forwarders logs to find those fail to connect to a target

© 2 0 2 0 S P L U N K I N C .
Incomplete Output Lists Create No Errors
Indexer A Indexer B Indexer C
Forwarder
We must search for the absence of connections to find this problem
Forwarders can only send data to targets that
are in their list.
This is very common when the indexer cluster
is increased in size.
Encourage customers to use indexer discovery
on forwarders so this never happens.
replication
Outputs:
B+C

© 2 0 2 0 S P L U N K I N C .
Do All Indexers Have the Same Forwarders?
index=_internal earliest=-24hrs latest=now sourcetype=splunkd TERM(eventType=connect_done)
TERM(group=tcpin_connections)
| stats count
by host sourceHost sourceIp
| stats
dc(host) as indexer_target_count
values(host) as indexers_connected_to
by sourceHost sourceIp
| eventstats max(indexer_target_count) as total_pool
| eval missing_indexer_count=total_pool-indexer_target_count
| where missing_indexer_count != 0
Search for forwarders logs to find those fail to connect to a target
This search assumes that
all indexers have the
same forwarders. With
multiple clusters and sites,
this might not be true

© 2 0 2 0 S P L U N K I N C .
Site Aware Forwarder Connections
index=_internal earliest=-24hrs latest=now sourcetype=splunkd TERM(eventType=connect_done) TERM(group=tcpin_connections)
| eval remote_forwarder=sourceHost." & ".if(sourceIp!=sourceHost,"(".sourceIp.")","")
| stats count as total_connections
by host remote_forwarder
| join host
[ search index=_internal earliest=-1hrs latest=now sourcetype=splunkd CMMaster status=success site*
| rex field=message max_match=64 "(?<site_pair>sited+,"[^"]+)"
| eval cluster_master=host
| fields + site_pair cluster_master
| fields - _*
| dedup site_pair
| mvexpand site_pair
| dedup site_pair
| rex field=site_pair "^(?<site_id>sited+),"(?<host>.*)"
| eventstats count as site_size by site_id cluster_master
| table site_id cluster_master host site_size]
| eval unique_site=site_id." & ".cluster_master
| chart
values(site_size) as site_size
values(host) as indexers_connected_to
by remote_forwarder unique_site
| foreach site*
[| eval "length_<<FIELD>>"=mvcount('<<FIELD>>')]
What sites forwarders connect to which site and what
is the coverage for that site?
Sub-search returns
indexer list
Sub-search
computes indexer
to site mapping

© 2 0 2 0 S P L U N K I N C .
What is Indexer Starvation?
When an indexer or an indexer pipeline
has periods when it is starved of data.
It will continue to receive replicated
data and be assigned primaries by the
CM to search replicated cold buckets.
It will not search hot buckets without
site affinity.
indexer indexer indexer
Forwarder
Indexers must receive data to participate in search
replication

© 2 0 2 0 S P L U N K I N C .
Smoke Test: Count Connections to Indexers
index=_internal earliest=-1hrs latest=now sourcetype=splunkd
TERM(eventType=connect_done) TERM(group=tcpin_connections)
| timechart limit=0 count minspan=31sec by host
This quick smoke test shows if
the indexers in the cluster have
obviously varying numbers of
incoming connections.
When you see banding it either a
smoking gun, or different
forwarder groups per site.
Indexer starvation is guaranteed if
there are not enough incoming
connections.
Fix by increase connections by increasing switching frequency and the
number of pipelines on forwarders
Nice and healthy

© 2 0 2 0 S P L U N K I N C .
Funnel Effect Reduces Connections
When an intermediate forwarder
aggregates multiple streams, it creates
a super giant forwarder.
You need to add more pipelines as
each pipeline creates a new TCP output
to the indexers.
Note that time division multiplexing
doesn’t mean 1 CPU = 1 pipeline,
unless it is running at maximum rate of
about 20 MB/s
Apply configuration with care.
Forwarder
FWDFWD FWD FWD FWDFWD

© 2 0 2 0 S P L U N K I N C .
How to Instrument and Tune IUFs
Forwarder
FWDFWD FWD FWD FWDFWD
Monitor out going connections
to understand cluster
coverage
Monitor CPU, load
average, pipeline utilization
and pipeline blocking
Find and reconfigure sticky
forwarders to use
EVENT_BREAKER
Monitor active channels to
avoid s2s protocol churn
Monitor incoming
connections to ensure
even distribution of
workload over pipelines
Use autoLBVolume
Enable forceTimeBasedAutoLB
Lengthen
timeBasedAutoLBFrequency
Add up to 16 pipelines but don’t
breach ~30% CPU
Monitor event delay to make
sure it doesn’t increase.
A well tuned intermediate forwarder can achieve 5%
event distribution across 100 indexers within 5mins

Best Practices for Forwarder Hierarchies

More Related Content

What's hot (20)

Similar to Best Practices for Forwarder Hierarchies (20)

More from Splunk (20)

Recently uploaded (20)

Best Practices for Forwarder Hierarchies