Accelerating Ceph with RDMA and NVMe-oF
1 likeâ˘3,693 views
The document discusses the integration of RDMA and NVMe over Fabrics (NVMe-oF) within the Ceph storage platform, highlighting performance benefits and future optimizations. RDMA is shown to improve Ceph's performance by up to 17% compared to TCP/IP, while also enhancing scalability without degrading write performance. The document concludes with a summary of findings and outlines next steps for optimizing RDMA networking components and leveraging NVMe-oF for cost-efficient storage solutions.
Accelerating Ceph with RDMA and NVMe-oF
- 1. 14th ANNUAL WORKSHOP 2018 ACCELERATING CEPH WITH RDMA AND NVME-OF Haodong Tang, Jian Zhang and Fred Zhang Intel Corporation {haodong.tang, jian.zhang, fred.zhang}@intel.com April, 2018
- 2. AGENDA ď§ Background and motivation ď§ RDMA as Ceph networking component ď§ RDMA as Ceph NVMe fabrics ď§ Summary & next step
- 4. CEPH INTRODUCTION ď§ References: http://ceph.com/ceph-storage, http://thenewstack.io/software-defined-storage-ceph-way, RADOS A software-based, reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes and lightweight monitors LIBRADOS A library allowing apps to directly access RADOS RGW A web services gateway for object storage Application RBD A reliable, fully distributed block device CephFS A distributed file system with POSIX semantics Host/VM Client ď§ Scalability â CRUSH data placement, no single POF ď§ Replicates and re-balances dynamically ď§ Enterprise features â snapshots, cloning, mirroring ď§ Most popular block storage for Openstack use cases ď§ Commercial support from Red Hat ď§ Open-source, object-based scale-out storage ď§ Object, Block and File in single unified storage cluster ď§ Highly durable, available â replication, erasure coding ď§ Runs on economical commodity hardware ď§ 10 years of hardening, vibrant community
- 5. CEPH PERFORMANCE PROFILING ď§ CPU is uneven distributed. ď§ CPU tend to be the bottleneck for 4K random write and 4K random read. ď§ Ceph networking layer consumes 20%+ CPU of the totally CPU used by Ceph in 4K random read workload. AsyncMsg(~22 - ~24%) * This picture is from the Boston OpenStack Summit
- 6. MOTIVATION ď§ RDMA is a direct access from the memory of one computer into that of another without involving either oneâs operating system. ď§ RDMA supports zero-copy networking(kernel bypass). ⢠Eliminate CPUs, memory or context switches. ⢠Reduce latency and enable fast messenger transfer. ď§ Potential benefit for ceph. ⢠Better Resource Allocation â Bring additional disk to servers with spare CPU. ⢠Lower latency - generated by ceph network stack.
- 7. RDMA AS CEPH NETWORKING COMPONENT
- 8. RDMA IN CEPH ď§ XIO Messenger. ⢠Based on Accelio, seamlessly supporting RDMA. ⢠Scalability issue. ⢠Merged to Ceph master three years ago, no support for now. ď§ Async Messenger. ⢠Async Messenger is compatible with different network protocol, like Posix, RDMA and DPDK. ⢠Current RDMA implementation supports IB protocol. HW Ethernet NIC(RNIC) NIC Driver Dispatcher Kernel Async Messenger IO Library RDMA Stack + OFA Verb APInetwork stack event driver event driver event driver workers pool dispatch queue event driver event driver event driver workers pool same pool IB Link IB Transport IB Transport APIkernel bypass
- 9. RDMA OVER ETHERNET ď§ Motivation ⢠Leverage RDMA to improve performance (low CPU, low latency). ⢠Leverage Intel RDMA NIC to accelerate Ceph. ⢠RDMA over Ethernet provide is one of the most convenient and practical way for datacenter running Ceph over TCP/IP. ď§ To-do ⢠Need introduce rdma-cm library. HW Ethernet NIC(RNIC) NIC Driver Dispatcher Kernel Async Messenger IO Library RDMA Stack + OFA Verb APInetwork stack event driver event driver event driver workers pool dispatch queue event driver event driver event driver workers pool same pool TCP/IP MPA DDP RDMAPkernel bypass
- 10. IMPLEMENTATION DETAILS ď§ Current implementation for Infiniband in Ceph: ⢠Connection management: Self-implemented TCP/IP based RDMA connection management ⢠RDMA verbs: RDMA send, RDMA recv ⢠Queue pairs: Shared receive queue (SRQ) ⢠Completed Queue: All queue pair share one completed queue ď§ iWARP protocol needs: ⢠Connection management: RDMA-CM based RDMA connection management ⢠Queue pairs: centralized memory pool for recv queue (RQ)
- 11. The networking component protocol between OSD node and client node can be changed. We compared the Ceph performance w/ TCP/IP and it w/ RDMA protocol. BENCHMARK METHODOLOGY â SMALL SCALE CPU SKX Platform (112 cores) Memory 128 GB NIC 10 GbE IntelÂŽ Ethernet Connection X722 with iWARP Disk distribution 4x P3700 as OSD drive, 1x Optane as DB driver Software configuration CentOS 7, Ceph Luminous (dev) FIO version 2.17 MON OSD OSD OSD OSD OSD OSD OSD OSD FIO FIO FIO FIO FIO FIOClient Node OSD Node
- 12. CEPH PERFORMANCE â TCP/IP VS RDMA â 1X OSD NDOE ď§ Ceph w/ iWARP delivers higher 4K random write performance than it with TCP/IP. ď§ Ceph w/ iWARP generates higher CPU Utilization. ⢠Ceph w/ iWARP consumes more user level CPU. ⢠Ceph w/ TCP/IP consumes more system level CPU. 0% 5% 10% 15% 20% 25% 30% 0 20000 40000 60000 80000 100000 120000 QD=1 QD=2 QD=4 QD=8 QD=16 QD=32 QD=64 IOPS Ceph Performance Comparison - RDMA vs TCP/IP - 1x OSD Node 4K Random Write RDMA Cluster 4K Random Write IOPS TCP/IP Cluster 4K Random Write IOPS RDMA Cluster CPU Utilization TCP/IP Cluster CPU Utilization 0 5 10 15 20 25 TCP/IP Cluster CPU Utilzation RDMA Cluster CPU Utilization Ceph CPU Comparison - RDMA vs TCP/IP - QD=64 4K Random Write usr sys iowait soft 18.7% 20.5% 4.5% 2.8%
- 13. We scale the OSD node to verify the RDMA protocol scale-out ability. BENCHMARK METHODOLOGY â LARGER SCALE CPU SKX Platform (72 cores) Memory 128 GB NIC 10 GbE IntelÂŽ Ethernet Connection X722 with iWARP Disk distribution 4x P4500 as OSD/DB drive Software configuration Ubuntu 17.10, Ceph Luminous (dev) FIO version 2.12 MON OSD OSD OSD OSD OSD OSD OSD OSD FIO FIO FIO FIO FIO FIOClient Node OSD Node FIO FIO FIO FIO FIO FIO MON OSD OSD OSD OSD OSD OSD OSD OSD
- 14. CEPH PERFORMANCE â TCP/IP VS RDMA â 2X OSD NODES 0% 10% 20% 30% 40% 50% 60% 70% 0 20000 40000 60000 80000 100000 120000 QD=1 QD=2 QD=4 QD=8 QD=16 QD=32 CPUUtilization IOPS Ceph Performance Comparison - RDMA vs TCP/IP - 2x OSD Nodes 4K Random Write RDMA Cluster 4K Random Write IOPS TCP/IP Cluster 4K Random Write IOPS RDMA Cluster CPU Utilization TCP/IP Cluster CPU Utilization 0 200 400 600 800 1000 1200 1400 1600 1800 QD=1 QD=2 QD=4 QD=8 QD=16 QD=32 IOPS Ceph Performance Comparison - RDMA vs TCP/IP 4K Random Write IOPS / CPU Utilization% RDMA Cluster 4K Random Write IOPS/CPU utilization% TCP/IP Cluster 4K Random Write IOPS/CPU utilization% ď§ Ceph w/ iWARP delivers up to 17% 4K random write performance benefit than it w/ TCP/IP. ď§ Ceph w/ iWARP is more CPU efficient. 13% 17% 7% 0% 13% 3% Higher is better
- 15. CEPH PERFORMANCE â TCP/IP VS RDMA â 3X OSD NODES ď§ Ceph node scaling out: RDMA vs TCP/IP - 48.7% vs 50.3% ď¨ scale out well. ď§ When QD is 16, Ceph w/ RDMA shows 12% higher 4K random write performance. 82409 122601 72289 108685 0 20000 40000 60000 80000 100000 120000 140000 2x OSD nodes 3x OSD nodes IOPS Ceph Performance Comparison - RDMA vs TCP/IP â QD=16 Scale-out performance RDMA Cluster 4K Random Write IOPS TCP/IP Cluster 4K Random Write IOPS 13% 12%48.7% 50.3%
- 16. PERFORMANCE RESULT DEEP ANALYSIS CPU Profiling ď§ Two polling thread: Ceph Epoll based Async driver thread + RDMA polling thread. ď§ Not really zero-copy: thereâs one copy from RDMA recv buffer to Ceph Async driver buffer.
- 17. RDMA AS CEPH NVME FABRICS
- 18. RDMA AS CEPH NVME FABRICS ď§ NVMe is a new specification optimized for NAND flash and next-generation solid-state storage technologies. ď§ NVMe over Fabrics enables access to remote NVMe devices over multiple network fabrics. ď§ Supported fabrics ď§ RDMA â InfiniBand, IWARP, RoCE ď§ Fiber Channel ď§ TCP/IP ď§ NVMe-oF benefits ď§ NVMe disaggregation. ď§ Delivers performance of remote NVMe on-par with local NVMe.
- 19. RDMA AS CEPH NVME FABRICS ď§ Baseline and comparison ď§ The baseline setup used local NVMe. ď§ The comparison setup attaches remote NVMe as OSD data drive. ď§ 6x 2T P3700 are among 2x Storage nodes. ď§ OSD nodes attach the 6x P3700 over RoCE V2 fabric. ď§ Set NVMe-oF CPU offload on target node. ď§ Hardware configuration ď§ 2x Storage nodes, 3x OSD nodes, 3x Client nodes. ď§ 6x P3700 (800 GB U.2), 3x Optane (375 GB) ď§ 30x FIO processes worked on 30x RBD volumes. ď§ All these 8x servers are BRW, 128 GB memory, Mellanox Connect-X4 NICs. Optane Ceph OSD P3700 P3700 client Optane Ceph OSD P3700 P3700 client Optane Ceph OSD P3700 P3700 client Ceph Client RBD RBD RBD RBD FIO Ceph Client RBD RBD RBD RBD FIO Ceph Client RBD RBD RBD RBD FIO TCP/IP Optane Ceph OSD P3700 P3700 NVMf client Optane Ceph OSD P3700 P3700 NVMf client Optane Ceph OSD P3700 P3700 NVMf client Ceph Client RBD RBD RBD RBD FIO Ceph Client RBD RBD RBD RBD FIO Ceph Client RBD RBD RBD RBD FIO P3700 P3700 P3700 Ceph Target P3700 P3700 P3700 Ceph Target RDMA TCP/IP
- 20. EXPECTATION BEFORE POC ď§ Expectations and questions before POC. ď§ Expectations: According to the benchmark from the first part, weâre expecting ď§ on-par 4K random write performance. ď§ on-par CPU utilization on NVMe-oF host node. ď§ Questions: ď§ How many CPU will be used on NVMe-oF target node ? ď§ How is the behavior of tail latency(99.0%) latency with NVMe-oF ? ď§ Does NVMe-oF influence the Scale-out ability of Ceph ?
- 21. RDMA AS CEPH NVME FABRICS ď§ On-par 4K random write performance ď§ Running Ceph with NVMe-oF brings <1% CPU overhead on target node. ď§ CPU is not the bottleneck on the host node. Client side performance comparison 0 50000 100000 150000 200000 250000 QD=1 QD=2 QD=4 QD=8 QD=16 QD=32 QD=64 QD=128 IOPS 4K Random Write - Ceph over NVMf vs Ceph over local NVMe 4K RW - Ceph with Local NVMe 4K RW - Ceph with NVMe-oF CPU Utilization on OSD Node CPU Utilization on Target Node
- 22. CEPH TAIL LATENCY ď§ When QD is higher than 16, Ceph with NVMe-oF shows higher tail latency (99%). ď§ When QD is lower than 16, Ceph with NVMe-oF on-par with Ceph over local NVMe. 0 50 100 150 200 250 300 350 QD=1 QD=2 QD=4 QD=8 QD=16 QD=32 QD=64 QD=128 ms Tail Latency Comparison - Ceph over NVMf vs Ceph over local NVMe 4K RW - Ceph with Local NVMe 4K RW - Ceph with NVMe-oF lower is better
- 23. RDMA AS CEPH NVME FABRICS ď§ Running Ceph over NVMe-oF didnât limit the Ceph OSD node scaling out. ď§ For 4K random write/read, the maximum ratio of 3x nodes to 2x nodes is 1.47, closing to 1.5 (ideal value). Scaling out performance 0 50000 100000 150000 200000 QD=1 QD=2 QD=4 QD=8 QD=16 QD=32 QD=64 QD=128 IOPS Scaling Out Testing - Ceph over NVMf 4K Random Write 2xnodes 3xnodes 0 200000 400000 600000 800000 1000000 QD=1 QD=2 QD=4 QD=8 QD=16 QD=32 QD=64 QD=128 IOPS Scaling Out Testing - Ceph over NVMf 4K Random Read 2xnodes 3xnodes 1.3 1.35 1.4 1.45 1.5 QD=1 QD=2 QD=4 QD=8 QD=16 QD=32 QD=64 QD=128 Performance Comparison- Ceph over NVMf 4K Random Write, 3x nodes/2x nodes 1.2 1.25 1.3 1.35 1.4 1.45 1.5 QD=1 QD=2 QD=4 QD=8 QD=16 QD=32 QD=64 QD=128 Performance Comparison - Ceph over NVMf 4K Random Write, 3x nodes/2x nodes
- 24. SUMMARY
- 25. SUMMARY & NEXT-STEP ď§ Summary ď§ RDMA is critical for future Ceph AFA solutions. ď§ Ceph with RDMA messenger provides up to ~17% performance advantage over TCP/IP. ď§ Ceph with RDMA messenger shows great scale-our ability. ď§ As network fabrics, RDMA performs well in Ceph NVMe-oF solutions. ď§ Running Ceph on NVMe-oF does not appreciably degrade Ceph write performance. ď§ Ceph with NVMe-oF brings more flexible provisioning and lower TCO. ď§ Next-step ď§ Ceph RDMA networking component optimization based on previous analysis. ď§ leverage NVMe-oF with the high density storage node for lower TCO.
- 26. LEGAL DISCLAIMER & OPTIMIZATION NOTICE Optimization Notice Intelâs compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice revision #20110804 ď§ Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit www.intel.com/benchmarks. ď§ INFORMATION IN THIS DOCUMENT IS PROVIDED âAS ISâ. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO THIS INFORMATION INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. ď§ Copyright Š 2018, Intel Corporation. All rights reserved. Intel, Pentium, Xeon, Xeon Phi, Core, VTune, Cilk, and the Intel logo are trademarks of Intel Corporation in the U.S. and other countries.
- 27. 14th ANNUAL WORKSHOP 2018 THANK YOU! Haodong Tang, Jian Zhang and Fred Zhang Intel Corporation {haodong.tang, jian.zhang, fred.zhang}@intel.com April, 2018