SF-TAP: Scalable and Flexible Traffic Analysis Platform (USENIX LISA 2015)
Download as PDF, PPTX4 likes1,238 views
The document presents sf-tap, a scalable and flexible traffic analysis platform designed for high-speed application-level traffic analysis on commodity hardware. It allows for programming in multiple languages and features modular architecture, flow abstraction, and horizontal and CPU core scalability. Performance evaluations demonstrate its efficiency in analyzing high bandwidth traffic and handling multiple connections per second.
![Related Work
6
BPF [USENIX ATC 1993]
netmap [USENIX ATC 2012]DPDK
pcap
GASPP [USENIX ATC 2014]
SCAP [IMC 2012]
libnids libprotoident
nDPI
l7-filter
(low level traffic capture)
(flow oriented analyzer) (application traffic detector)
SF-TAP
+ modularity and scalability](https://image.slidesharecdn.com/sftap-usenix-lisa2015-v2-151112153819-lva1-app6891/85/SF-TAP-Scalable-and-Flexible-Traffic-Analysis-Platform-USENIX-LISA-2015-6-320.jpg)
![Performance Evaluation (1)
16
forwarding performance of SF-TAP cell incubator
Mpps
0
4
8
12
16
fragment size (bytes)
64 128 256 512 1024
ideal (1) α->γ (2) α->β (3) α->β (3) α->γ
e 14: Forwarding Performance of Cell Incubator
risks are incurr
Host-based I
it is not suitab
used in today’s
power. Therefo
operating with
support the futu
face of the flow
implementation
protocols.
7 Related
Wireshark [38]
Figure 11: CPU Load of Flow Abstractor versus Traffic
Volume
Figure 16 shows the CPU loads of the 15th CPU. At
5.95 Mpps, the load average was approximately 50%, but
at 10.42 Mpps, the loads were close to 100%. More-
over, at 14.88 Mpps, CPU resources were completely
consumed. This limitation in forwarding performance
was probably caused by the bias, which in turn was due
o the flow director [10] of Intel’s NIC and its driver. The
flow director cannot currently be controlled by user pro-
grams on FreeBSD; thus, it causes bias depending on net-
work flows. Note that the fairness regarding RSS queues
s simply an implementation issue and is benchmarked
or future work.
Finally, the memory utilization of the cell incubator
depends on the memory allocation strategy of netmap.
The current implementation of the cell incubator requires
approximately 700 MB of memory to conduct the exper-
ments.
6 Discussion and Future Work
Figure 12: Physical Memory Usage of Flow Ab
(10K CPS)
1 GbE x 12
10 GbE x 2
α β
γ
cell incubator
Figure 13: Experimental Network of Cell Incu
6.1 Performance Improvements
We plan to improve the performance of the flow
tor in three aspects.
(1) The UNIX domain socket can be replaced
other mechanism such as a memory-mapped file o
memory attach [6]; however, these mechanisms
suitable for our approach, which abstracts flows
Thus, new mechanisms for high-performance m
passing, such as the zero-copy UNIX domain so
zero-copy pipe, should be studied.
(2) The flow abstractor currently uses the mallo
to γ to β to β and γ
α
β
γ](https://image.slidesharecdn.com/sftap-usenix-lisa2015-v2-151112153819-lva1-app6891/85/SF-TAP-Scalable-and-Flexible-Traffic-Analysis-Platform-USENIX-LISA-2015-16-320.jpg)
More Related Content
![[Webinar Slides] Programming the Network Dataplane in P4](https://cdn.slidesharecdn.com/ss_thumbnails/p4-webinar-intro-demo-oct2015-chang-151106220822-lva1-app6891-thumbnail.jpg?width=560&fit=bounds)
Similar to SF-TAP: Scalable and Flexible Traffic Analysis Platform (USENIX LISA 2015) (20)
![Strategic Picks — Prioritising the Right Agentic Use Cases [2/6]](https://cdn.slidesharecdn.com/ss_thumbnails/session-2-250902135931-4c97717a-thumbnail.jpg?width=560&fit=bounds)
![From Curiosity to ROI — Cost-Benefit Analysis of Agentic Automation [3/6]](https://cdn.slidesharecdn.com/ss_thumbnails/session-3-250903141006-a3917e1a-thumbnail.jpg?width=560&fit=bounds)
SF-TAP: Scalable and Flexible Traffic Analysis Platform (USENIX LISA 2015)
- 1. SF-TAP: Scalable and Flexible Traffic Analysis Platform Running on Commodity Hardware 1 Yuuki Takano, Ryosuke Miura, Shingo Yasuda Kunio Akashi, Tomoya Inoue NICT, JAIST (Japan) in USENIX LISA 2015
- 2. Table of Contents 1. Motivation 2. Related Work 3. Design of SF-TAP 4. Implementation of SF-TAP 5. Performance Evaluation 6. Conclusion 2
- 3. Motivation (1) • Programmable application level traffic analyzer • We want … • to write traffic analyzers in any languages such as Python, Ruby, C++, for many purposes (IDS/ IPS, forensic, machine learning). • **not** to write codes handling TCP stream reconstruction (quite complex). • modularity for many application protocols. 3
- 4. Motivation (2) • High speed application level traffic analyzer • We want … • to handle high bandwidth traffic. • to handle high connections per second. • horizontal and CPU core scalable analyzer. 4
- 5. Motivation (3) • Running on Commodity Hardware • We want … • open source software. • not to use expensive appliances. 5
- 6. Related Work 6 BPF [USENIX ATC 1993] netmap [USENIX ATC 2012]DPDK pcap GASPP [USENIX ATC 2014] SCAP [IMC 2012] libnids libprotoident nDPI l7-filter (low level traffic capture) (flow oriented analyzer) (application traffic detector) SF-TAP + modularity and scalability
- 7. High-level Architecture of SF-TAP 7 CPU CPU CPU CPU Flow Abstractor CPU CPU CPU CPU Flow Abstractor CPU CPU CPU CPU Flow Abstractor CPU CPU CPU CPU Flow Abstractor Cell Incubator The Internet SF-TAP Cell SF-TAP Cell SF-TAP Cell SF-TAP Cell Intra Network Core ScalingCore Scaling Core Scaling Core Scaling Horizontal Scaling Analyzer Analyzer Analyzer Analyzer Analyzer Analyzer Analyzer Analyzer Analyzer Analyzer Analyzer Analyzer Analyzer Analyzer Analyzer Analyzer 10GbE 10GbE
- 8. Design Principle (1) • Flow Abstraction • abstract flows by application level protocols • provide flow abstraction interfaces like /dev, /proc or BPF • for multiple programming languages • Modular Architecture • separate analyzing and capturing logic • easily replace analyzing logic 8
- 9. Design Principle (2) • Horizontal Scalable • analyzing logic tends to require many computer resources • volume effect should solve the problem • CPU Core Scalable • both analyzing and capturing logic should be core scalable for efficiency 9
- 10. Design of SF-TAP (1) 10 NW I/F HTTP I/F TLS I/F Flow Abstractor Flow Classifier TLS Analyzer HTTP Analyzer HTTP Proxy TCP and UDP Handler filter and classifier rule L7 Loopback I/F DB Forensic IDS/IPS etc... Application Protocol Analyzer etc...TCP Default I/F UDP Default I/F Analyzer PlaneAbstractor Plane Capturer Plane SF-TAP Cell Incubator Flow Identifier Flow Separator Separator Plane separated traffic SF-TAP Cell L3/L7 Sniffer SSL Proxy etc... other SF-TAP cells IP Packet Defragmenter L2 Bridge mirroring traffic Packet Forwarder IP Fragment Handler defined 4 planes Analyzer Plane application level analyzers Forensic, IDS/IPS, etc… Abstractor Plane flow abstraction Separator Plane flow separation Capturer Plane traffic capturing (ordinary tech.) (users of SF-TAP implements here) (we implemented) (we implemented)
- 11. Design of SF-TAP (2) SF-TAP Cell Incubator 11 SF-TAP Cell Incubator Flow Separator separated traffic other SF-TAP cells L2 Bridge Packet Forwarder IP Fragment Handler Packet Forwarder layer 2 bridge layer 2 frame capture IP Fragment Handler handle fragmented packets Flow Separator separate flows to multiple Ifs
- 12. Design of SF-TAP (3) SF-TAP Flow Abstractor 12 NW I/F HTTP I/F TLS I/F Flow Abstractor Flow Classifier TCP and UDP Handler filter and classifier rule L7 Loopback I/F TCP Default I/F UDP Default I/F Flow Identifier IP Packet Defragmenter TCP and UDP Handler Flow Identifier IP Packet Defragmenter reconstruct TCP flows identify flows by IP and port nothing to do for UDP defragment IP packets if needed
- 13. Design of SF-TAP (4) SF-TAP Flow Abstractor 13 NW I/F HTTP I/F TLS I/F Flow Abstractor Flow Classifier TCP and UDP Handler filter and classifier rule L7 Loopback I/F TCP Default I/F UDP Default I/F Flow Identifier IP Packet Defragmenter Flow Classifier classify flows by regular expressions output to abstraction IFs
- 14. Implementation • SF-TAP cell incubator • C++11 • it uses netmap, available on FreeBSD • SF-TAP flow abstractor • C++11 • it uses pcap or netmap • available on Linux, *BSD, and MacOS • Source Code • https://github.com/SF-TAP • License • 3-clauses BSD 14 (updated from the paper)
- 15. Performance Evaluation (1) 15 Figure 8: Total Memory Usage of HTTP Analyzer Figure 9: Packet Drop against CPS packet drop against connections per second (pcap) 4K 10K 50K
- 16. Performance Evaluation (1) 16 forwarding performance of SF-TAP cell incubator Mpps 0 4 8 12 16 fragment size (bytes) 64 128 256 512 1024 ideal (1) α->γ (2) α->β (3) α->β (3) α->γ e 14: Forwarding Performance of Cell Incubator risks are incurr Host-based I it is not suitab used in today’s power. Therefo operating with support the futu face of the flow implementation protocols. 7 Related Wireshark [38] Figure 11: CPU Load of Flow Abstractor versus Traffic Volume Figure 16 shows the CPU loads of the 15th CPU. At 5.95 Mpps, the load average was approximately 50%, but at 10.42 Mpps, the loads were close to 100%. More- over, at 14.88 Mpps, CPU resources were completely consumed. This limitation in forwarding performance was probably caused by the bias, which in turn was due o the flow director [10] of Intel’s NIC and its driver. The flow director cannot currently be controlled by user pro- grams on FreeBSD; thus, it causes bias depending on net- work flows. Note that the fairness regarding RSS queues s simply an implementation issue and is benchmarked or future work. Finally, the memory utilization of the cell incubator depends on the memory allocation strategy of netmap. The current implementation of the cell incubator requires approximately 700 MB of memory to conduct the exper- ments. 6 Discussion and Future Work Figure 12: Physical Memory Usage of Flow Ab (10K CPS) 1 GbE x 12 10 GbE x 2 α β γ cell incubator Figure 13: Experimental Network of Cell Incu 6.1 Performance Improvements We plan to improve the performance of the flow tor in three aspects. (1) The UNIX domain socket can be replaced other mechanism such as a memory-mapped file o memory attach [6]; however, these mechanisms suitable for our approach, which abstracts flows Thus, new mechanisms for high-performance m passing, such as the zero-copy UNIX domain so zero-copy pipe, should be studied. (2) The flow abstractor currently uses the mallo to γ to β to β and γ α β γ
- 17. Other Features • L7 Loopback interface for encapsulated flows • Load balancing mechanism for application protocol analysers • Separating and mirroring modes of SF-TAP cell incubator • See more detains in our paper 17
- 18. Conclusion • We proposed SF-TAP for application level traffic analysis. • SF-TAP has following features. • flow abstraction • running on commodity hardware • modularity • scalability • We showed SF-TAP has achieved high performance in our experiments. 18