PLNOG20 - Paweł Małachowski - Stress your DUT–wykorzystanie narzędzi open source w testach urządzeń sieciowych

Stress your DUT
Using open source tools for network device dataplane testing.
Paweł Małachowski, 2018.03.20
@pawmal80

Agenda
 introduction
 software packet generators overview
 Packet Of Death story
 redGuardian regression testing internals

Basics
 RFC 2544 „Benchmarking Methodology for Network Interconnect Devices”
Frame sizes to be used on Ethernet 64, 128, 256, 512, 1024, 1280, 1518
 RFC 6815 „Use on Production Networks Considered Harmful”
 RFC2889 „Benchmarking Methodology for LAN Switching Devices”
 IMIX concept

Example metrics
 throughput (bps, pps)
 zero packet loss
 latency
 connection/flow setup rate

Example: packet switching/routing
 „Mellanox Spectrum vs. Broadcom StrataXGS Tomahawk 25GbE&100GbE
Performance Evaluation” (Tolly, 2016)
http://www.mellanox.com/related-docs/products/tolly-report-
performance-evaluation-2016-march.pdf
 „Cisco Integrated Services Routers—Performance Overview” (2010)
https://supportforums.cisco.com/legacyfs/online/legacy/7/2/6/134627-
white_paper_c11_595485.pdf

Example: routing loopkup performance on Linux
 https://vincent.bernat.im/
en/blog/2017-ipv4-route-
lookup-linux
 https://vincent.bernat.im/
en/blog/2017-
performance-progression-
ipv4-route-lookup-linux
 great in-depth analysis

Example factors
 random dst IP
 MSS clamping?
 FIB size (Trie vs. DIR-m-n)
 port interaction
 VM interaction (NFV)
 do you have baseline?

Software packet generation: TX methods
API comment
socket (RAW, DGRAM)
pcap classic, mature, available everywhere
in-kernel Linux pktgen
AF_PACKET (V4+ZC) XDP cooperation
kernel bypass (DPDK,
Netmap, PF_RING ZC, Snabb)
fastest possible, available only on some platforms,
frequently require additional compiling/patching

Classic generators
 available OOTB or easy to install
 mature, well documented
 pcap(3) based
 „fast enough” in some cases

hping3
 command-line oriented TCP/IP packet assembler and analyzer
 notable options: flood, spoofing, addres/port randomization
 1 Mpps easy to achieve
 similar tools: nping (nmap)

hping3
% hping3 --syn 127.0.0.1 --destport ++31337
HPING 127.0.0.1 (lo 127.0.0.1): S set, 40 headers + 0 data bytes
len=40 ip=127.0.0.1 ttl=64 DF id=46879 sport=31337 flags=RA seq=0 win=0 rtt=7.7 ms
^C
--- 127.0.0.1 hping statistic ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 3.3/4.8/7.7 ms
% hping3 --syn 127.0.0.1 --destport ++31337 --flood

tcpreplay
 tools for editing and replaying network traffic
 pcap(3) based, Netmap support
 idea: record your UDP traffic, replay it against tested service
 https://github.com/appneta/tcpreplay

tcpreplay
% tcpreplay -i eth2 -K -t -l 100000 ./test/test.pcap
File Cache is enabled
Actual: 14100000 packets (6270400000 bytes) sent in 9.07 seconds.
Rated: 645928366.4 Bps, 5167.42 Mbps, 1452473.52 pps
Flows: 37 flows, 3.81 fps, 14000000 flow packets, 100000 non-flow
Statistics for network device: eth2
Attempted packets: 14100000
Successful packets: 14100000
Failed packets: 0
Truncated packets: 0
Retried packets (ENOBUFS): 0
Retried packets (EAGAIN): 0

Scapy
 Python-based packet manipulation program/library
 packet (and layers) are Python objects, e.g. Ether(), IP()
 read/write pcap file/interface
 slow but useful for automated packet crafting

Scapy
$ sudo scapy
>>> ICMP()
<ICMP |>
>>> ICMP(seq=0xda).show()
###[ ICMP ]###
type= echo-request
code= 0
chksum= None
id= 0x0
seq= 0xda
>>> pkts=Ether() / IP(src="10.0.0.0/24", dst="192.168.1.1") / ICMP()
>>> pkts
<Ether type=0x800 |<IP frag=0 proto=icmp src=Net('10.0.0.0/24') dst=192.168.1.1 |<ICMP |>>>
>>> sendp(pkts, iface="eth1")
Sent 256 packets.

Ostinato
 „Wireshark in Reverse”
 craft packets with GUI
 send at specific rate
 DPDK acceleration
(prototype only, 2014)
 https://ostinato.org/

Ostinato
 click & learn
 available as GNS3 and
VIRL image

Accelerated generators
 kernel bypassing for maximum performance
 rough edges
 non–trivial to install, tune (core to RX/TX queue mapping,
driver and PCIe tuning) and operate
 poor man’s alternative for hardware packet generators

Snabb packetblaster
 Snabb (LuaJIT) based
 able to push 20x10Gbps with little CPU usage
 tight TX loop over preloaded packets
 https://github.com/snabbco/snabb/tree/master/src/program/packetblaster
% packetblaster replay myfile.cap 0000:01:00.0

DPDK pktgen
 classic DPDK-based packet generator
 CLI-oriented, multiple „pages”
 good for „naive” packet flooding
 http://pktgen-dpdk.readthedocs.io/,
http://dpdk.org/browse/apps/pktgen-dpdk/refs/

DPDK pktgen
 load pcap and randomize fields
 Lua scripts (API/control only)
% pktgen --socket-mem=128,16384 -w
0000:82:00.0,txq_inline=128 -l 0,12-23 -n 2 -- -N
-T -m "[12:13-23].0"

MoonGen
 „Scriptable High-Speed Packet Generator”
 DPDK + LuaJIT based
 craft your packets in Lua!
 nice, scientific approach
 https://github.com/emmericp/MoonGen
% moongen-simple start udp-simple:0:0:rate=1000mbit/s,ratePattern=poisson

T-Rex
 „generates L4-7 traffic based on pre-processing and smart replay of
real traffic templates”
 DPDK-based
 feature rich
 https://trex-tgn.cisco.com/,
https://github.com/cisco-system-traffic-generator

T-Rex
 IMIX floods
 application traffic
 TCP connections
% t-rex-64 -f cap2/imix_64.yaml
-c 4 -m 1 -d 100 -l 1000

WARP17
 „Lightweight solution for generating high volumes of session-based traffic
with high setup rates”
 DPDK-based
 http://warp17.net/,
https://github.com/Juniper/warp17
% warp17 -l 0-7 -n 2 -m 16384 -w 0000:01:00.1 -- --qmap-default max-q --
cmd-file=atds.cfg

WARP17
 server &
client
 TCP sessions
setup
 DUT as
middlebox

Packet Of Death
 repeatable fails of nightly regression
tests
 network card hangs, only machine
reboot helps
 WTF?

Packet of Death
 git log tests/
 facepalm
 vendor releases firmware upgrade

redGuardian testing framework
Goals
 functional regression
 local/pcap-based
 remote (real NICs)
 performance regression
Tools
 Python
 pytest
 Scapy
 DPDK pktgen

Example testcase
def _conf(cfg):
cfg.add_target_v4('0.0.0.0/0')
@with_config(_conf)
class TestForward:
def test_ipv4_fragment_beyond_end(self, tester):
"""Fragmented packets with sum of fragment offset and IP length exceeding 65535 are dropped"""
src = '1.2.3.4'
dst = '10.0.0.1'
tester.run(PASS << RAND_ETH << [
PASS << IP(src=src, dst=dst, proto='udp', id=1, frag=8189) / Payload(3), # =65535
DROP << IP(src=src, dst=dst, proto='udp', id=1, frag=8189) / Payload(4), # =65536
])

„Expected” packet actions
Action Expected behaviour
DROP silent drop
PASS passthrough
DEC_TTL decrement TTL
DELAY wait (e.g. for state expiration)
FRAGMENT perform IP fragmentation
STRIP_VLAN remove 802.1Q tag
TRUNCATE truncate packet (packet sampling)
…

Example performance testcase
TTL1 = RAND_ETH / IP(src=RandIP(), dst=RandIP(), ttl=1) / TCP()
class TestPerfInvalid(TestPerfBase):
def test_ttl1_drop(self, perf_tester, cfg):
cfg.add_target_v4('0.0.0.0/0')
perf_tester.perf_pcap = TTL1
self.run(perf_tester, cfg, 'ttl1_drop')
$ tests/run.sh --perf tests/perf/test_perf_invalid.py::TestPerfInvalid::test_ttl1_drop
platform linux -- Python 3.5.3, pytest-3.4.0, py-1.5.2, pluggy-0.6.0 --
/home/pawmal/git/dpapp/dataplane/tests/venv/bin/python
perf_cfg: pktgen_default.cfg
rootdir: /home/pawmal/git/dpapp/dataplane, inifile:
plugins: xdist-1.22.0, forked-0.2
collected 1 item
tests/perf/test_perf_invalid.py::TestPerfInvalid::test_ttl1_drop PASSED [100%]
PERF ttl1_drop rx_median 14.730; tx_median 0.000; rx_mean 14.620; tx_mean 0.000; gen_rx 0.000; gen_tx
14.813; gen_rx_Mbps 0; gen_tx_Mbps 9954

Performance regression tracking
 multiple
performance
scenarios
 compare Mpps
between nightly
builds
 1 core E5-2695 v4
2.10GHz, random
tuples

NIC benchmarking
 https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_perfo
rmance_report.pdf
 https://fast.dpdk.org/doc/perf/DPDK_17_11_Mellanox_NIC_
performance_report.pdf
 consider these „theoretical maximum”

Testing frameworks
 DPDK Test Suite: https://dpdk.org/doc/dts/gsg/index.html
 fd.io/VPP Continuous System Integration and Testing (CSIT):
https://docs.fd.io/csit/master/doc/

Summary
 theoretical vs. real life vs. IMIX
 understand your DUT internals
 networking product development without automated testing
is impossible

PLNOG20 - Paweł Małachowski - Stress your DUT–wykorzystanie narzędzi open source w testach urządzeń sieciowych

More Related Content

What's hot (20)

Similar to PLNOG20 - Paweł Małachowski - Stress your DUT–wykorzystanie narzędzi open source w testach urządzeń sieciowych (20)

Recently uploaded (20)

PLNOG20 - Paweł Małachowski - Stress your DUT–wykorzystanie narzędzi open source w testach urządzeń sieciowych