Data Storage & I/O Performance: Solving I/O Slowdown: The "Noisy Neighbor" Problem
1 like707 views
The document discusses the challenges of input/output (I/O) performance in shared high-speed parallel filesystems and highlights new Lustre features that address these issues, such as Progressive File Layout (PFL) and Data on Metadata (DOM). These advancements enable more effective data placement and isolation of 'noisy neighbor' applications, thereby improving performance for various workloads. Overall, the improvements aid in optimizing throughput and IOPS while maintaining sequential performance.
Data Storage & I/O Performance: Solving I/O Slowdown: The "Noisy Neighbor" Problem
- 1. © 2018 Cray Inc. 1 Solving I/O the Slowdown: The "Noisy Neighbor" Problem Rice Oil and Gas Conference 2019 John Fragalla Principal Engineer Cray, Inc.
- 2. © 2018 Cray Inc. 2 • Today’s I/O challenge with shared I/O • New Lustre Features enabling Flash to improve Shared Application Performance • Performance results isolating “Noisy Neighbor” Applications • Summary Agenda
- 3. © 2018 Cray Inc. 3 Today’s I/O Challenge • When multiple users share a high speed parallel filesystem, ”bad applications” will effect “good application” performance • Bad applications: Lots of Small Files, Random Small I/O, Unaligned I/O • Good applications: Stream large I/O, Sequential performance, Aligned I/O • Recent features available in Lustre help automate I/O isolation and placement with transparent use of Flash and HDD Devices in a Single Namespace • Progressive File Layout with Lustre Storage Pools • Data on Metadata • Distributed Namespace (DNE) 2 – Clustered Metadata
- 4. © 2018 Cray Inc. 4 Hybrid File System Architecture CN CN CN CN CN CN CN CN CN CN CN CN CN CN CN CN CN CN CN CN HDD HDD HDD HDD HDD HDD HDD HDD SSD SSD SSD SSD SSD SSD SSD SSD Parallel File System Single Namespace Compute Cluster Flash Tier : • Optimize for throughput and IOPs - $/GB/sec. • Improved Performance for Intermediate Results • Small/random I/O performance improved High Performance HDD Tier: • Optimize Throughput/Capacity - $/GB/Sec • Optional Flash/Cache to accelerate small block IO within the HDD Tier HDD HDD HDD HDD HDD HDD HDD HDD HDD HDD HDD HDD Capacity HDD Tier: • Optimize cost - $/GB • Lower performance, longer term data retention MDS MDS MDS MDS Scalable MDS Flash Tier : • Large number of Inode Support per FS • Improved Metadata Operations • Improved Small I/O Latency
- 5. © 2018 Cray Inc. 5 • Progressive File Layouts (PFL) • Optimized striping based on file size • Layout changes at specific thresholds • Can locate components on specific pools • Fixed amount of file on flash, the rest on disk Lustre | Flexibility and Usability
- 6. © 2018 Cray Inc. 6 • Lustre Pools historically was used for debugging to isolate performance issues, for example, for a subset of OSTs or OSS Nodes. • Now, with PFL, Lustre Storage Pools can be a powerful tool to create automatic data placement on different storage medias • Flash Pool • High Performing Disk Pool • Slow Performing Disk Tier (e.g. focus on capacity) Lustre Storage Pools are more relevant Now
- 7. © 2018 Cray Inc. 7 Lustre | Improved Small File Performance • Data on Metadata • Ideal for small file workloads • File data stored directly on metadata storage • Lower communication overhead for data access • Scales with Distributed Namespace (DNE) • Avoids contention by not placing small files on OSTs
- 8. © 2018 Cray Inc. 8 Lustre | Data on Metadata and PFL • Leverage DoM and PFL for more flexible solution • Small files land on MDT Component • Medium files land on flash with larger files growing to disk (for example) • Compatible with Progressive File Layouts
- 9. © 2018 Cray Inc. 9 DNE Phase 2 • Allows a user to spread a single large directory across multiple MDTs using the DNE striped directory feature • Note due to some overhead, this should only be done to very large directories with file counts in +50K range. DNE phase 1 DNE phase 2
- 10. © 2018 Cray Inc. 10 System Setup for Benchmarks Hardware: • 4 Flash MDTs with RAID-10 • 2 Flash IOPS Optimized OSTs with RAID-10 • 4 GridRAID OSTs (Parity De- Clustered RAID-6 equivalent data protection) • Up to 64 Client nodes (FDR Connectivity) • EDR InfiniBand Non-Blocking Fabric Software: • Lustre 2.11.0 clients and server • CentOS Linux release 7.5 (server and client) • Spectre/Meltdown-enabled kernels on Clients, S/M disabled on Server • Client: 3.10.0-862.el7.x86_64 • Server: 3.10.0-693.21.1.x3.1.9.x86_64
- 11. © 2018 Cray Inc. 11 LUSTRE PFL STREAMING PERFORMANCE Flash MDTs -> HDD OST Tier (DoM) 0 5,000 10,000 15,000 20,000 25,000 30,000 No DoM DoM=64K DoM=256K DoM=1024K DoM=4096K Performance(MB/sec) Progressive File Layout Small Component Size Write Mean Read Mean Progressive File Layout maintains peak performance for streaming workloads We want no change in performance across various sizes
- 12. © 2018 Cray Inc. 12 LUSTRE PFL NOISY NEIGHBOR ISOLATION Lustre HDD OST File 1 File 2 File 3 File 4 File Small File Workload Streaming Workload HDD OST File 1 File 2 File 3 File 4 File Small File Workload Streaming Workload Flash OST or MDT Two Competing Workloads on Same Resources Two Workloads Separated using PFL
- 13. © 2018 Cray Inc. 13 12,000 14,000 16,000 18,000 20,000 22,000 24,000 None None (1MB) 1024K (1MB) MB/s 1 MB FILES Write Mean Read Mean 12,000 14,000 16,000 18,000 20,000 22,000 24,000 None None (4MB) 1024K (4MB) 4096K (4MB) MB/s 4 MB FILES Write Mean Read Mean LUSTRE PFL NOISY NEIGHBOR ISOLATION Flash Tier (OST or DoM with MDTs) -> HDD OST Tier PFL ISOLATION OF IOPS FROM STREAMING IMPROVES PERFORMANCE Baseline Interfered Isolated Baseline Isolated Interfered X-Axis Legend PFL Size on Flash (Noisy Neighbor File Size)
- 14. © 2018 Cray Inc. 14 • New Lustre Features such as PFL, DoM, and DNE2 help improve mixed I/O performance on high speed shared parallel filesystem • Transparent data placement on Flash MDTs and/or OSTs and HDDs for various I/O sizes to optimize throughput and IOPS • Isolate small files or small I/Os from streaming I/O to solve the “Noisy Neighbor” slow down for sequential performance Summary
- 15. THANK YOU Q U E S T I O N S ? cray.com