Transparent eBPF Offload: Playing Nice with the Linux Kernel
3 likes•1,156 views
The document provides an overview of eBPF (extended Berkeley Packet Filter) and XDP (express data path), including their architecture, programming model, and use cases in enhancing kernel functionality and network performance. eBPF allows developers to run sandboxed programs in the kernel for tasks like load balancing and security, while XDP enables high-speed packet processing before they are attached to sockets. It also discusses the programming model, how to create maps for data interaction, and various hooks available in the kernel for BPF.
Transparent eBPF Offload: Playing Nice with the Linux Kernel
- 1. ©2016 Open-NFP 1 BPF and XDP Explained Nic Viljoen
- 2. ©2017 Open-NFP 2 Objectives of the Webinar Give user a basic understanding of the architecture of eBPF ▪ What is it ▪ The programming model ▪ The kernel hooks Give user a basic understanding of XDP ▪ What is it/Where is it ▪ How to use it (beginner level!) ▪ How to offload it
- 3. ©2016 Open-NFP 3 What is eBPF? eBPF is a simple way to extend the functionality of the kernel at runtime ▪ Effectively a small kernel based machine ▪ 10 64bit registers ▪ 512 byte stack ▪ Data structures known as maps (unlimited size) ▪ 4K BPF instructions (Bytecode) ▪ Verifier to ensure kernel safe ▪ no loops, not more than 4K insns, not more than 64 maps etc… ▪ Can be JITed to ensure maximum performance
- 4. ©2016 Open-NFP 4 Used Within Hyperscale-Not a Toy! Those who have publically stated they are using BPF or are planning to use BPF include ▪ Facebook-Load Balancing, Security ▪ Netflix-Network Monitoring ▪ Cilium Project ▪ Cloudflare-Security ▪ OVS-Virtual Switching Due to its upstream safety and kernel support BPF provides a safe, flexible and scalable networking tool
- 5. ©2016 Open-NFP 5 The Programming Model LLVM is used to compile from supported languages ▪ C ▪ Go ▪ P4 When Programs are loaded ▪ Verifier is called-ensure safety ▪ Program is JITed-ensure perf ▪ Can also be offloaded ▪ nfp_bpf_jit upstream LL VM NFP verifier.c bpf_prog.go bpf_prog.elf bpf syscall USER JIT nfp_bfp_jit.c Host CPU KERNEL HARDWARE bpf_prog.p4 bpf_prog.c
- 6. ©2016 Open-NFP 6 Maps-What They Are Maps are key value stores ▪ Can be accessed from kernel or user space ▪ Used for interaction between kernel and user space programs Number of different types of maps ▪ Used for interaction between kernel and user space programs bpf_user.c bpf_kern.c Map
- 7. ©2017 Open-NFP 7 Maps-How to use them Creating Maps ▪ Option 1: create map with syscall ▪ bpf(BPF_MAP_CREATE, &bpf_attr, sizeof(bpf_attr)) ▪ Option 2: define a struct bpf_map_def with an elf section __attribute__ SEC(“maps”)-also uses syscall! Option 1 Option 2 THIS IS AN OVERSIMPLIFICATION
- 8. ©2017 Open-NFP 8 eBPF Bytecode: Quick Overview eBPF Bytecode: op:8, dst_reg:4, src_reg:4, off:16, imm:32 ▪ op code is divided into the sections ▪ Operation code (4bits) e.g BPF_MOV, BPF_JNE ▪ Source bit (1 bit) BPF_X (use src_reg and dst_reg) or BPF_K (use dst_reg and 32 bit imm) ▪ instruction class (3 bits) e.g BPF_ALU, BPF_ALU64, BPF_JMP ▪ BPF_MOV | BPF_X | BPF_ALU64, 0x6, 0x1, 0x0000, 0x00000000 ▪ Move contents of register 1 to register 6 ▪ BPF_JNE | BPF_K | BPF_JMP, 0x1, 0x0, 0x0011, 0x00008100 ▪ Jump 11 insns forward-can also jump backwards-if contents of register 1 is not equal to 0x00008100
- 9. ©2017 Open-NFP 9 BPF Kernel Hooks Many hooks with different purposes ▪ kprobes ▪ socket filters-tcpdump-old school! ▪ seccomp ▪ netfilter (new) ▪ TC ▪ XDP(no skb-super fast!) XDP will be our focus for the rest of this talk
- 10. ©2017 Open-NFP 10 XDP BPF hook in the driver ▪ Allows for high speed processing before skb is attached to packet ▪ Currently 4 return codes: XDP_ABORT, XDP_DROP, XDP_PASS, XDP_TX ▪ XDP_REDIRECT in the pipeline ▪ Usecases include DDoS protection and load balancing ▪ Includes maximum of 256 bytes of prepend ▪ Metadata is just pointers to start of packet and end
- 11. ©2017 Open-NFP 11 Program Example (xdp1_kern.c) Simple drop example ▪ Note the use of standard header infrastructure ▪ Associated user space program maintaining a set of counters ▪ I am not going to go through line by line-for more detail check out Andy and Jesper’s awesome tutorial-in links ▪ Will come back to this example later on… This can be found in the recent (4.8+) kernels at linux/samples/bpf
- 12. ©2017 Open-NFP 12 Optimizing XDP A simple checklist-not comprehensive! ▪ Ensure BPF JIT is enabled ▪ Pin queues to interfaces ▪ Set ringsize to an optimal level for your NIC and application ▪ To gain some idea of your NIC’s driver based XDP performance check simple XDP_DROP and XDP_TX programs ▪ Many people use single core performance as a reasonable benchmark ▪ To do this use the ethtool -X command ▪ You will NOT get the simple program performance if you build something complex (Duh)
- 13. ©2017 Open-NFP 13 Offloading XDP Netronome have upstreamed the initial version of the nfp_bpf_jit ▪ More to come!
- 14. ©2017 Open-NFP 14 Offload Architecture user space kernel space BPF syscall ● program ● type (sk filter, kprobe, cls, xdp) ● license ● ... verifier fd host JIT tc TC cls_bpf modification XDP ctrl offload object fd, skip_* flags verification fd, skip_* flags driver RX TXXDP ndo setup tc HW JIT / translator stats & maps BPF prog
- 15. ©2017 Open-NFP 15 References Kernel Docs: https://www.kernel.org/doc/Documentation/networking/filter.txt Initial XDP Presentation: https://github.com/iovisor/bpf-docs/blob/master/ Express_Data_Path.pdf More Docs: http://prototype-kernel.readthedocs.io/en/latest/README.html Andy and Jesper’s Talk: https://netdevconf.org/2.1/slides/apr7/gospodarek- Netdev2.1-XDP-for-the-Rest-of-Us_Final.pdf Reading List: https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf/ Search: google.com :)
- 16. ©2016 Open-NFP 16 ANY QUESTIONS? Thanks!