Accelerating analytics workloads with Alluxio data orchestration and Intel® Optane™ persistent memory
0 likes332 views
The document discusses test results from running the TPC-DS benchmark on Spark SQL with different Alluxio caching layers: DRAM, Intel Optane persistent memory (PMem) using Storage over App Direct mode, and Intel Optane SSDs. PMem and Optane SSD caching completed more queries than DRAM-only caching and had significantly faster query times. While PMem and Optane SSD caching performed similarly in CPU usage and hardware metrics, PMem caching had some advantages like higher caching capacity and lower query times for some workloads. More analysis is needed to understand Alluxio-specific metrics and performance in cloud environments.
Accelerating analytics workloads with Alluxio data orchestration and Intel® Optane™ persistent memory
- 1. Alluxio Community Day Accelerating analytics workloads with Alluxio data orchestration and Intel® Optane™ persistent memory Ginger Gilsdorf Software Engineer
- 2. 2 Alluxio Community Day June 24, 2021 ▪ The hardware • Xeon, PMem, Optane SSD ▪ The software • Alluxio, TPC-DS, Spark ▪ The results ▪ The analysis Hardware without software just generates heat… Agenda Software without hardware is just 1s and 0s… -Doug Fisher
- 3. 3 Alluxio Community Day June 24, 2021 ▪ The hardware • Xeon, PMem, Optane SSD ▪ The software • Alluxio, TPC-DS, Spark ▪ The results ▪ The analysis Hardware without software just generates heat… Agenda Software without hardware is just 1s and 0s… -Doug Fisher
- 4. Alluxio Community Day June 24, 2021 4 Intel® Xeon® Scalable Processors Ice Lake is here! 2nd Gen 3rd Gen Cascade Lake Ice Lake 2019 April 2021 28 40 Up to 6 channels @2933 MT/s Up to 8 channels @3200 MT/s 100 series (Apache Pass) 200 series (Barlow Pass) Gen 3 Gen 4 1.00 1.461 1Please visit www.intel.com/3gen-xeon-config and use the corresponding performance number 125 to access full system configuration and performance detail. Cloud availability coming…
- 5. Alluxio Community Day June 24, 2021 5 Intel Optane family ▪ PMem 200 series • 32% higher bandwidth • Up to 16% lower power • Up to 4 TB per socket1 • Operating modes: Memory, App Direct, Storage over App Direct, Mixed Intel® Optane™ Technology Intel Optane Memory Media Intel Memory and Storage Controllers Intel Interconnect IP Intel Software *Images may not reflect all current offerings. See https://www.intel.com/content/www/us/en/architecture-and-technology/intel-optane-technology.html for details. IntelOptane persistent memory (PMem) • DIMM form factor • 128, 256, 512 GB modules • PMem; non-volatile memory (NVM) • 200 series launched with 3rd Gen Xeon® Scalable Processors 14 TB PMem per socket using 8x 512GB DIMMs (leaves 8x slots for DRAM).
- 6. Alluxio Community Day June 24, 2021 6 Intel Optane family Intel® Optane™ Technology Intel Optane Memory Media Intel Memory and Storage Controllers Intel Interconnect IP Intel Software *Images may not reflect all current offerings. See https://www.intel.com/content/www/us/en/architecture-and-technology/intel-optane-technology.html for details. IntelOptane persistent memory • DIMM form factor • 128, 256, 512 GB modules • PMem; non-volatile memory (NVM) • 200 series launched with 3rd Gen Xeon® Scalable Processors Intel Optane DC SSDs P4800X and P4801X Series P5800X Series Intel Optane Memory M10, H10, H20 Series
- 7. 7 Alluxio Community Day June 24, 2021 ▪ The hardware • Xeon, PMem, Optane SSD ▪ The software • Alluxio, TPC-DS, Spark ▪ The results ▪ The analysis Hardware without software just generates heat… Agenda Software without hardware is just 1s and 0s… -Doug Fisher
- 8. Alluxio Community Day June 24, 2021 8 Alluxio & Intel *Source: https://www.alluxio.io/intel/ Multi-year collaboration, including 2nd & 3rd Gen Xeon, PMem, and Optane DC SSDs
- 9. Alluxio Community Day June 24, 2021 9 TPC-DS ▪ Enterprise-class decision support benchmark ▪ Measures performance of SQL- based big data systems ▪ Models typical queries and data maintenance tasks ▪ Performance metric: query response time and throughput • Scale factor options: 1, 3, 10, 30, 100 TB Examples: • For a given year, month, and store manager calculate the total store sales of any combination of all brands. • Select the top revenue- generating products bought by out-of-zip code customers for a given year, month, and manager. Source: http://www.tpc.org/tpcds/
- 10. Alluxio Community Day June 24, 2021 10 TPC-DS with Spark SQL ▪ Enterprise-class decision support benchmark ▪ Measures performance of SQL- based big data systems ▪ Models typical queries and data maintenance tasks ▪ Performance metric: query response time and throughput • Scale factor options: 1, 3, 10, 30, 100 TB Source: http://www.tpc.org/tpcds/ ▪ Apache Spark provides the distributed data processing engine ▪ Version 2.0 supports all 99 TPC- DS queries ▪ Convenient repo to setup the TPC-DS toolkit, generate data, and run the benchmark: • https://github.com/IBM/spark-tpc- ds-performance-test
- 11. 11 Alluxio Community Day June 24, 2021 ▪ The hardware • Xeon, PMem, Optane SSD ▪ The software • Alluxio, TPC-DS, Spark ▪ The results ▪ The analysis Hardware without software just generates heat… Agenda Software without hardware is just 1s and 0s… -Doug Fisher
- 12. Alluxio Community Day June 24, 2021 12 Test setup ▪ All on-premises testing ▪ Disaggregated compute & storage • Compute running on Ice Lake • Storage on Cascade Lake ▪ Alluxio caching layers tested: Compute: • 2x Xeon Platinum 8368 CPU • 38 cores each @ 2.40GHz • Spark 2.4.7 • Alluxio 2.4.1 Compute: • 2x Xeon Platinum 8368 CPU • 38 cores each @ 2.40GHz • Spark 2.4.7 • Alluxio 2.4.1 Storage: • 2x Xeon Platinum 8280 CPU • 28 cores each @ 2.70GHz • Minio Capacity/Compute node (GB) 768 1024 PMem + 192 DRAM 800 Similar cost ▪ Scale factor 3 TB; Compressed data ~ 2.1 TB *Storage over App Direct
- 13. Alluxio Community Day June 24, 2021 13 Test results: Queries completed ▪ 99 TPC-DS queries total ▪ Some timed out (returned 0 rows) DRAM PMem (SoAD) Optane SSD 1536 2048 1600 58 80 80 445 51 53 N/A 88% 88% 926.81 316.44 322.20 N/A 211.20 822.41 95.23 39.03 35.04 Samples →
- 14. 14 Alluxio Community Day June 24, 2021 ▪ The hardware • Xeon, PMem, Optane SSD ▪ The software • Alluxio, TPC-DS, Spark ▪ The results ▪ The analysis Hardware without software just generates heat… Agenda Software without hardware is just 1s and 0s… -Doug Fisher
- 15. Alluxio Community Day June 24, 2021 15 Caching layer differences ▪ Alluxio DRAM layer: • Shares DRAM with workload • Less capacity for caching • More remote data access needed ▪ Alluxio PMem & Optane SSD: • Cached data separate from DRAM • More capacity; less remote access ▪ Clear impact to performance Capacity/Compute node (GB) 768 1024 PMem + 192 DRAM 800 *Storage over App Direct
- 16. Alluxio Community Day June 24, 2021 16 PMem (SoAD) vs. Optane SSD ▪ Both using block storage ▪ PMem on memory bus; ▪ Optane SSD on PCIe ▪ Many similarities in CPU usage & hardware metrics Average CPU utilization/ Operating frequency PMem (SoAD) Optane SSD 14% / 3.22 13% / 3.22 20% / 3.25 21% / 3.23 22% / 3.21 19% / 3.21 26% / 3.21 20% / 3.21 PMem Optane SSD ▪ Both measured DDR rate at 3200 MT/sec
- 17. Alluxio Community Day June 24, 2021 17 PMem (SoAD) vs. Optane SSD ▪ A few differences… ▪ System-level metrics capturing both TPC-DS workload & Alluxio data orchestration Query 15 PMem (SoAD) Optane SSD 211.20 822.41 0.82 1.10 59,317 50,827 Ranges from 0 to 1,552 0 (N/A) 142 165
- 18. Alluxio Community Day June 24, 2021 18 Conclusions & next steps ▪ Capacity matters! • Impact to performance varies ▪ PMem offers higher capacities than similar-costing DRAM ▪ Optane SSD is also a good option in many cases ▪ Follow up work: • More hardware metric analysis – Alluxio specific • More testing, especially in cloud
- 19. 19 Alluxio Community Day June 24, 2021 ▪ Alluxio & Intel Partnership ▪ alluxio.io/intel/ ▪ Intel Optane Technology ▪ intel.com/optane ▪ Intel Developer Zone ▪ software.intel.com/persistent- memory ▪ Persistent Memory Programming ▪ pmem.io Questions? Resources
- 20. 20