Introduction to DPDK
Download as PPTX, PDF10 likes6,684 views
DPDK is a set of drivers and libraries that allow applications to bypass the Linux kernel and access network interface cards directly for very high performance packet processing. It is commonly used for software routers, switches, and other network applications. DPDK can achieve over 11 times higher packet forwarding rates than applications using the Linux kernel network stack alone. While it provides best-in-class performance, DPDK also has disadvantages like reduced security and isolation from standard Linux services.
Introduction to DPDK
- 1. DPDK Data Plane Development Kit Ariel Waizel [email protected]
- 2. 2Confidential Who Am I? • Programmer for 12 years – Mostly C/C++ for cyber and networks applications. – Some higher languages for machine learning and generic DevOps (Python and Java). • M.sc in Information System Engineering (a.k.a machine learning). • Currently I’m a solution architect at Contextream ( HPE) – The company specializes in software defined networks for carrier grade companies (Like Bezek, Partner, etc.). – Appeal: Everything is open-source! – I work with costumers on end-to-end solutions and create POCs. • Gaming for fun.
- 3. 3Confidential Let’s Talk About DPDK • What is DPDK? • Why use it? • How good is it? • How does it work?
- 4. 4Confidential The Challenge • Network application, done in software, supporting native network speeds. • Network speeds: – 10 Gb per NIC (Network Interface Card). – 40 Gb per NIC. – 100 Gb? • Network applications: – Network device implemented in software (Router, switch, http proxy, etc.). – Network services for virtual machines (Openstack, etc.). – Security. protect your applications from DDOS.
- 5. 5Confidential What is DPDK? • Set of UIO (User IO) drivers and user space libraries. • Goal: forward network packet to/from NIC (Network Interface Card) from/to user application at native speed: – 10 or 40 Gb NICs. – Speed is the most (only?) important criteria. – Only forwards the packets – not a network stack (But there’re helpful libraries and examples to use). • All traffic bypasses the kernel (We’ll get to why). – When a NIC is controlled by a DPDK driver, it’s invisible to the kernel. • Open source (BSD-3 for most, GPL for Linux Kernel related parts).
- 6. 6Confidential Why DPDK Should Interest Kernel Devs? • Bypassing the kernel is importance because of performance - Intriguing by itself. – At the very least, know the “competition”. • DPDK is a very light-weight, low level, performance driven framework – Makes it a good learning ground for kernel developers, to learn performance guidelines.
- 7. 7Confidential Why bypassing the kernel is a necessity. Why Use DPDK?
- 8. 8Confidential 10Gb – Crunching Numbers • Minimum Ethernet packet: 64 bytes + 20 preamble. • Maximum number of pps: 14,880,952 – 10^10 bps /(84 bytes * 8) • Minimum time to process a single packet: 67.2 ns • Minimum CPU cycles: 201 cycles on a 3 Ghz CPU (1 Ghz -> 1 cycle per ns).
- 9. 9Confidential Time (ish) per Operation (July 2014) Time (Expected)Operation 32 nsCache Miss 4.3 nsL2 cache access 7.9 nsL3 cache access 8.25 ns (16.5 ns for unlock too)“LOCK” operation (like Atomic) 41.85 ns (75.34 for audit-syscall)syscall 80 ns for alloc + freeSLUB Allocator (Linux Kernel buffer allocator) 77 nsSKB (Linux Kernel packet struct) Memory Manager 48 ns minimum, between 58 to 68 ns averageQdisc (tx queue descriptor) Up to several Cache Misses.TLB Miss * Red Hat Challenge 10Gbit/s Jesper Dangaard Brouer et al. Netfilter Workshop July 2014. Conclusion: Must use batching (Send/rcv several packet together at the same time, amortize costs)
- 10. 10Confidential Time Per Operation – The Big No-Nos Average TimeOperation Micro seconds (1000 ns)Context Switch Micro secondsPage Fault Conclusion: Must pin CPUs and pre-allocate resources
- 11. 11Confidential Benchmarks at the end. As of Today: DPDK can handle 11X the traffic Linux Kernel Can!
- 12. 12Confidential How It Works. DPDK Architecture
- 13. 13Confidential DPDK – What do you get? • UIO drivers • PMD per hardware NIC: – PMD (Poll Mode Driver) support for RX/TX (Receive and Transmit). – Mapping PCI memory and registers. – Mapping user memory (for example: packet memory buffers) into the NIC. – Configuration of specific HW accelerations in the NIC. • User space libraries: – Initialize PMDs – Threading (builds on pthread). – CPU Management – Memory Management (Huge pages only!). – Hashing, Scheduler, pipeline (packet steering), etc. – High performance support libraries for the application.
- 14. 14Confidential DPDK APP DPDK APP From NIC to Process – Pipe Line modelNIC RX Q RX Q PMD PMD TX Q TX Q DPDK APP RING RINGIgb_uio * igb_uio is the DPDK standard kernel uio driver for device control plane
- 15. 15Confidential From NIC to Process – Run To Completion ModelNIC RX Q RX Q TX Q TX Q DPDK APP + PMD DPDK APP + PMDIgb_uio
- 16. 16Confidential Software Configuration • C Code – GCC 4.5.X or later. • Required: – Kernel version >= 2.6.34 – Hugetablefs (For best performance use 1G pages, which require GRUB configuration). • Recommended: – isolcpus in GRUB configuration: isolates CPUs from the scheduler. • Compilation – DPDK applications are statically compiled/linked with the DPDK libraries, for best performance. – Export RTE_SDK and RTE_TARGET to develop the application from a misc. directory • Setup: – Best to use tools/dpdk-setup.sh to setup/build the environment. – Use tools/dpdk-devbind.py o -- status let’s you see the available NICs and their corresponding drivers. o -bind let’s you bind a NIC to a driver. Igb-uio, for example. – Run the application with the appropriate arguments.
- 17. 17Confidential Software Architecture • PMD drivers are just user space pthreads that call specific EAL functions – These EAL functions have concrete implementations per NIC, and this costs couple of indirections. – Access to RX/TX descriptors is direct. – Uses UIO driver for specific control changes (like configuring interrupts). • Most DPDK libraries are not thread-safe. – PMD drivers are non-preemptive: Can’t have 2 PMDs handle the same HW queue on the same NIC. • All inner-thread communication is based on librte_ring. – A mp-mc lockless non-resizable queue ring implementation. – Optimized for DPDK purposes. • All resources like memory (malloc), threads, descriptor queues, etc. are initialized at the start.
- 20. 20Confidential Code Example – Basic Forwarding • Main Function: DPDK Init Get all available NICS (binded with igb_uio) Initialize packet buffers Initialize NICs.
- 21. 21Confidential Code Example – port_init NIC init: Set number of queues Rx queue init: Set packet buffer pool for queue * Uses librte_ring to be thread safe Tx queue init: No need for buffer pool Start Getting Packets
- 22. 22Confidential Code Example – lcore_main PMD
- 23. 23Confidential Igb_uio • For Intel Gigabit NICs. – Simple enough to work for most NICs, nonetheless. • Basically: – Calls pci_enable_device. – Enables bus mastering on the device (pci_set_master). – Requests all BARs and mas them using ioremap. – Setups ioports. – Sets the dma mask to 64-bit. • Code to support SRIOV and xen.
- 24. 24Confidential rte_ring • Fixed sized, “lockless”, queue ring • Non Preemptive. • Supports multiple/single producer/consumer, and bulk actions. • Uses: – Single array of pointers. – Head/tail pointers for both producer and consumer (total 4 pointers). • To enqueue (Just like dequeue): – Until successful: o Save in local variable the current head_ptr. o head_next = head_ptr + num_objects o CAS the head_ptr to head_next – Insert objects. – Until successful: o Update tail_ptr = head_next + 1 when tail_ptr == head_ptr • Analysis: – Light weight. – In theory, both loops are costly. – In practice, as all threads are cpu bound, the amortized cost is low for the first loop, and very unlikely at the second loop.
- 25. 25Confidential rte_mempool • Smart memory allocation • Allocate the start of each memory buffer at a different memory channel/rank: – Most applications only want to see the first 64 bytes of the packet (Ether+ip header). – Requests for memory at different channels, same rank, are done concurrently by the memory controller. – Requests for memory at different ranks can be managed effectively by the memory controller. – Pads objects until gcd(obj_size,num_ranks*num_channels) == 1. • Maintain a per-core cache, bulk requests to the mempool ring. • Allocate memory based on: – NUMA – Contiguous virtual memory (Means also contiguous physical memory for huge pages). • Non- preemptive. – Same lcore must not context switch to another task using the same mempool.
- 26. 26Confidential Linux kernel Vs. DPDK Benchmarks
- 27. 27Confidential Linux Kernel Benchmarks, single core Feb. 2016July 2014Benchmark 14.8 Mpps4 MppsTX 12 Mpps (experimental)6.4 MppsRX (Dump at driver) 2 Mpps1 MppsL3 Forwarding (RX+Filter+TX) 12 Mpps6 MppsL3 forwarding (Multi Core) * Red Hat The 100Gbit/s Challenge Jesper Dangaard Brouer et al. DevConf Feb. 2016.
- 28. 28Confidential DPDK Benchmarks (March 2016) Multi CoreSingle CoreBenchmark Linear Increase22 MppsL3 forwarding (PHY-PHY) Linear Increase11 MppsSwitch Forwarding (PHY-OVS-PHY) Near Linear Increase3.4 MppsVM Forwarding (PHY-VM-PHY) Linear Increase2 MppsVM to VM * Intel Open Network Platform Release 2.1 Performance Test Report March 2016. • All tests with: • 4X40Gb ports • E5-2695 V4 2.1Ghz Processor • 16X1GB Huge Pages, 2048X2MB Huge Pages
- 29. 29Confidential DPDK Benchmarks Figures PHY-PHY PHY-OVS-PHY VM-VM
- 30. 30Confidential DPDK Pros and Cons
- 31. 31Confidential DPDK Advantages • Best forwarding performance solution to/from PHY/process/VM to date. – Best single core performance. – Scales: linear performance increase per core. • Active and longstanding community (from 2012). – DPDK.org – Full of tutorials, examples and complementary features. • Active popular products – OVS-DPDK is the main solution for high speed networking in openstack. – 6WIND. – TRex. • Great virtualization support. – Deploy at the host of the guest environment.
- 32. 32Confidential DPDK Disadvantages • Security • Isolated ecosystem: – Hard to use linux kernel infrastructure (While there are precedents). • Requires modified code in applications to use: – DPDK processes use a very specific API. – DPDK application can’t interact transparently with Linux processes (important for transparent networking applications like Firewall , DDOS mitigation, etc.). o Solved for interaction with VMs by the vhost Library. • Requires Huge pages (XDP doesn’t).