How Coupang Leverages Distributed Cache to Accelerate ML Model Training
Alluxio Tech Talk Webinar Apr. 22, 2025 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Hyun Jung Baek (Staff Backend Engineer @ Coupang) Description Coupang is a leading e-commerce company in South Korea, with over 50,000 employees and $20+ billion in annual revenue. Coupang's AI platform team builds and manages a large-scale AI platform in AWS for machine learning engineers to train models that enhance and customize product search results and product recommendations for its 100+ million customers. As the search and recommendation models evolve, optimizing the underlying infrastructure for AI/ML workloads is essential for the e-commerce business. Coupang's platform team actively sought to improve their model training pipeline to boost machine learning engineers' productivity, publish models to production faster, and reduce operational costs. Coupang focused on addressing several key areas: - Shortening data preparation and model training time - Improving GPU utilization in training clusters in different regions - Reducing S3 API and egress costs incurred from copying large training datasets across regions - Simplifying the operational complexity of storage system management In this tech talk, Hyun Jung Baek, Staff Backend Engineer at Coupang, will share best practices for leveraging Alluxio to power search and recommendation model training infrastructure. Hyun will discuss: - How Coupang builds a world-class large-scale AI platform for machine learning engineers to deliver better search and recommendation models - How adding distributed caching to their multi-region AI infrastructure improves GPU utilization, accelerates end-to-end training time, and significantly reduces cross-region data transfer costs. - How to simplify platform operations and to easily deploy the same architecture to new GPU clusters.
How Coupang Leverages Distributed Cache to Accelerate ML Model Training
- 1. How Coupang Leverages Distributed Cache to Accelerate ML Model Training April 22, 2025 Hyun Jung Baek, Staff Backend Engineer @ Coupang
- 2. Coupang Confidential and Proprietary Coupang is a technology and Fortune 200 company listed on the New York Stock Exchange (NYSE: CPNG) that provides retail, restaurant delivery, video streaming, and fintech services to customers around the world under brands that include Coupang, Coupang Eats, Coupang Play and Farfetch. Coupang is a Technology and Fortune 200 Company (NYSE: CPNG)
- 3. Coupang Confidential and Proprietary Machine Learning Impacts Every Aspect of Commerce Experiences of Coupang Customers Product Catalog Search Pricing Robotics Inventory Fulfilment Meet Coupang’s Machine Learning Platform: https://medium.com/coupang-engineering/meet-coupangs-machine-learning-platform-cd00e9ccc172
- 4. Coupang Confidential and Proprietary Core offerings • Notebooks & ML Pipeline Authoring • Model Training • Model Inference • Monitoring & Observability Coupang’s ML Platform Overview Meet Coupang’s Machine Learning Platform: https://medium.com/coupang-engineering/meet-coupangs-machine-learning-platform-cd00e9ccc172
- 5. Coupang Confidential and Proprietary Both AWS Multi-Region & On-prem GPU Clusters ● Cloud GPU clusters across AWS Asia- Pacific & US regions ● On-prem data center (compute & storage) Requirements ● Resource efficiency ○ GPU utilization ● High I/O throughput ● Developer experience ● Cloud cost optimization Hybrid & Multi-Region Compute & Storage Due to GPU Shortage Meet Coupang’s Machine Learning Platform: https://medium.com/coupang-engineering/meet-coupangs-machine-learning-platform-cd00e9ccc172 Monitoring GPU utilization of Training cluster
- 6. Coupang Confidential and Proprietary Previous Architecture ap-region On Prem Local Storage GPU Training Cluster Data Copy Data Lake ap-region Local Storage GPU Training Cluster us-region
- 7. Coupang Confidential and Proprietary ● Required preparation step (copy and validation) before training jobs ○ Added day-long delay before training on a dataset ● Challenges in fully utilizing GPU resources across regions ○ Difficult to run overflow training jobs in a different region if local cluster is peaked, as the data may not be available or may exist in different paths ● Data Silos and Storage cost growing ● Operation overhead to maintain storage organized and under capacity ○ Required coordination across teams to manage and maintain local storage Challenges of the Previous Architecture
- 8. Coupang Confidential and Proprietary New Architecture with Distributed Cache ap-region Data Lake On Prem Distributed Cache GPU Training Cluster Only on Cache Miss ap-region GPU Training Cluster us-region Distributed Cache
- 9. Coupang Confidential and Proprietary Inside Distributed Cache Worker Pod Worker Pod Worker Pod etcd Pod etcd Pod etcd Pod FUSE Pod FUSE Pod FUSE Pod FUSE Pod Training Job Pod hostpath: /mnt/cache-fuse I/O Request Mount Table & Membership Distributed Cache Service Data Lake Cache Miss
- 10. Coupang Confidential and Proprietary ● Instant Data Availability ○ Eliminates lengthy data preparation ■ Training jobs can start immediately without waiting for data to be cached ○ Model developers can still pre-load datasets using the --skip-if-exists flag ■ If already cached, this step is a no-op ○ No coordination required across teams, simplifying the workflow ● Improve GPU Utilization Across Multi-Region ○ Maintains a consistent view of all data paths from the original data lake address, enabling seamless access across regions ○ During peak GPU hours, developers can submit training jobs to an overflow GPU cluster, unmodified, ensuring higher GPU utilization across multiple regions ● Faster Training Jobs ○ Provides higher performance compared to traditional HPC storage solutions (e.g., AWS FSx), significantly reducing training time and boosting productivity New Architecture: Benefits for Model Developers
- 11. Coupang Confidential and Proprietary ● Reduced Storage Costs & Operation Overhead ○ Avoids full capacity storage purchases by eliminating duplicate datasets from data lakes ■ Data lake (many PBs) vs cache capacity (TB to PB) ○ No coordination required for cache space cleanup ● Easy Expansion & Operation ○ Seamlessly scale architecture to new GPU clusters without complex reconfiguration ○ Fully managed with Kubernetes (K8s) for simplified deployment, scaling, and maintenance across environments New Architecture: Benefits for Platform Engineers
- 12. Coupang Confidential and Proprietary THANK YOU Copyright © 2024 Coupang, Inc. All rights reserved. All Coupang trademarks, Coupang logos and service marks displayed herein are property of Coupang, Inc. and/or its affiliates (collectively, "Coupang"), registered in the U.S. and other countries. Any other company mentioned herein is merely for identification purposes. Coupang acknowledges that the company name may be a registered trademark of the company and recognizes that any such trademark is owned solely and exclusively by such company. The information contained herein are based on the author, Hyun Jung Baek's own individual experience as an employee and are not representative of any views or opinions of Coupang. Coupang has not verified, and it makes no representation as to, the adequacy, fairness, accuracy, or completeness of any information contained herein.