Alluxio + Spark: Accelerating Auto Data Tagging in WeRide

0 likes•282 views

The document discusses the integration of Alluxio and Spark to enhance automatic data tagging for autonomous driving applications at WeRide. It highlights the benefits of improved data management and performance, showcasing tasks being completed up to 7 times faster. Future developments will focus on further integrations and leveraging GPU capabilities alongside Spark and Alluxio.

Software

Alluxio + Spark: Accelerating Auto Data Tagging
in WeRide
Feifei Cai, Hao Zhu
12/14/2021

• Introduction
• Alluxio Overview
• WeRide Use Case
• Future Work

• Feifei Cai
• Hao Zhu
• @WeRide
Introduction

• Fast I/O
• Simple
• Easy
Alluxio Overview
Image Source:alluxio.io

• Data Driven
• Storage
• Computing
Autonomous Driving Development
Data
Collection
Analytics
&
Selection
Data
Labeling
Model
Training
Simulation
&
Validation
Deploy

• Tags vs Labels
• Scenarios
• Data Selection
Data Tagging
Image Source:CARLA

• Hybrid Cloud
• Kubernetes
• Data Locality
Alluxio + Spark in Kubernetes

• MEM / SSD: 2TB / 8TB
• Tasks:7x faster
Test Results

• Integrate with more: Presto...
Future Work

• Spark + GPU + Alluxio (working on it)
Future Work
Image Source:NVIDIA

More Related Content

What's hot (20)

PDF

Building an open data platform with apache icebergAlluxio, Inc.

PDF

Optimizing Latency-sensitive queries for Presto at Facebook: A Collaboration ...Alluxio, Inc.

PDF

Best Practices for Using Alluxio with SparkAlluxio, Inc.

PDF

Speeding Up Atlas Deep Learning Platform with Alluxio + FluidAlluxio, Inc.

PDF

Presto on Alluxio Hands-On LabAlluxio, Inc.

PDF

Ultra Fast Deep Learning in Hybrid Cloud Using Intel Analytics Zoo & AlluxioAlluxio, Inc.

PDF

Building Fast SQL Analytics on Anything with Presto, AlluxioAlluxio, Inc.

PDF

High Performance Data Lake with Apache Hudi and Alluxio at T3GoAlluxio, Inc.

PDF

How to Develop and Operate Cloud Native Data Platforms and ApplicationsAlluxio, Inc.

PDF

How to teach your data scientist to leverage an analytics cluster with Presto...Alluxio, Inc.

PDF

Alluxio - Virtual Unified File System Alluxio, Inc.

PDF

Unified Data Access with GimelAlluxio, Inc.

PDF

Presto: Fast SQL-on-Anything Across Data Lakes, DBMS, and NoSQL Data StoresAlluxio, Inc.

PDF

Using Alluxio as a Fault Tolerant Pluggable Optimization Component to Compute...Alluxio, Inc.

PDF

The Practice of Presto & Alluxio in E-Commerce Big Data PlatformAlluxio, Inc.

PDF

Apache Iceberg - A Table Format for Hige Analytic DatasetsAlluxio, Inc.

PDF

Introducing the Hub for Data OrchestrationAlluxio, Inc.

PDF

Apache Iceberg Presentation for the St. Louis Big Data IDEAAdam Doyle

PDF

Data Orchestration for the Hybrid Cloud EraAlluxio, Inc.

PDF

Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & AlluxioAlluxio, Inc.

Building an open data platform with apache icebergAlluxio, Inc.

Optimizing Latency-sensitive queries for Presto at Facebook: A Collaboration ...Alluxio, Inc.

Best Practices for Using Alluxio with SparkAlluxio, Inc.

Speeding Up Atlas Deep Learning Platform with Alluxio + FluidAlluxio, Inc.

Presto on Alluxio Hands-On LabAlluxio, Inc.

Ultra Fast Deep Learning in Hybrid Cloud Using Intel Analytics Zoo & AlluxioAlluxio, Inc.

Building Fast SQL Analytics on Anything with Presto, AlluxioAlluxio, Inc.

High Performance Data Lake with Apache Hudi and Alluxio at T3GoAlluxio, Inc.

How to Develop and Operate Cloud Native Data Platforms and ApplicationsAlluxio, Inc.

How to teach your data scientist to leverage an analytics cluster with Presto...Alluxio, Inc.

Alluxio - Virtual Unified File System Alluxio, Inc.

Unified Data Access with GimelAlluxio, Inc.

Presto: Fast SQL-on-Anything Across Data Lakes, DBMS, and NoSQL Data StoresAlluxio, Inc.

Using Alluxio as a Fault Tolerant Pluggable Optimization Component to Compute...Alluxio, Inc.

The Practice of Presto & Alluxio in E-Commerce Big Data PlatformAlluxio, Inc.

Apache Iceberg - A Table Format for Hige Analytic DatasetsAlluxio, Inc.

Introducing the Hub for Data OrchestrationAlluxio, Inc.

Apache Iceberg Presentation for the St. Louis Big Data IDEAAdam Doyle

Data Orchestration for the Hybrid Cloud EraAlluxio, Inc.

Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & AlluxioAlluxio, Inc.

Similar to Alluxio + Spark: Accelerating Auto Data Tagging in WeRide (20)

PDF

Best Practice in Accelerating Data Applications with Spark+AlluxioAlluxio, Inc.

PDF

Improving Data Locality for Spark Jobs on Kubernetes Using AlluxioAlluxio, Inc.

PDF

Best Practices for Using Alluxio with Apache Spark with Gene PangSpark Summit

PDF

Best Practices for Using Alluxio with SparkAlluxio, Inc.

PDF

Spark Summit EU talk by Jiri SimsaSpark Summit

PDF

Spark Summit EU talk by Jiri SimsaAlluxio, Inc.

PDF

Flexible and Fast Storage for Deep Learning with Alluxio Alluxio, Inc.

PDF

Best Practices for Using Alluxio with Apache Spark with Cheng Chang and Haoyu...Databricks

PDF

Getting Started with Apache Spark and Alluxio for Blazingly Fast AnalyticsAlluxio, Inc.

PDF

Alluxio @ Uber Seattle MeetupAlluxio, Inc.

PDF

Best Practices for Using Alluxio with SparkAlluxio, Inc.

PDF

Effective Spark with Alluxio at Strata+Hadoop World San Jose 2017Alluxio, Inc.

PPTX

Spark Pipelines in the Cloud with Alluxio by Bin FanData Con LA

PDF

Getting Started with Alluxio + Spark + S3Alluxio, Inc.

PDF

CNCF Member Webinar: Improving Data Locality for Analytics Jobs on Kubernetes...Alluxio, Inc.

PPTX

Alluxio: Unify Data at Memory SpeedAlluxio, Inc.

PDF

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...Alluxio, Inc.

PDF

Accelerating Spark with KubernetesAlluxio, Inc.

PDF

AI/ML Infra Meetup | Maximizing GPU Efficiency : Optimizing Model Training wi...Alluxio, Inc.

PDF

Accelerating Spark Workloads in a Mesos Environment with AlluxioAlluxio, Inc.

Best Practice in Accelerating Data Applications with Spark+AlluxioAlluxio, Inc.

Improving Data Locality for Spark Jobs on Kubernetes Using AlluxioAlluxio, Inc.

Best Practices for Using Alluxio with Apache Spark with Gene PangSpark Summit

Best Practices for Using Alluxio with SparkAlluxio, Inc.

Spark Summit EU talk by Jiri SimsaSpark Summit

Spark Summit EU talk by Jiri SimsaAlluxio, Inc.

Flexible and Fast Storage for Deep Learning with Alluxio Alluxio, Inc.

Best Practices for Using Alluxio with Apache Spark with Cheng Chang and Haoyu...Databricks

Getting Started with Apache Spark and Alluxio for Blazingly Fast AnalyticsAlluxio, Inc.

Alluxio @ Uber Seattle MeetupAlluxio, Inc.

Best Practices for Using Alluxio with SparkAlluxio, Inc.

Effective Spark with Alluxio at Strata+Hadoop World San Jose 2017Alluxio, Inc.

Spark Pipelines in the Cloud with Alluxio by Bin FanData Con LA

Getting Started with Alluxio + Spark + S3Alluxio, Inc.

CNCF Member Webinar: Improving Data Locality for Analytics Jobs on Kubernetes...Alluxio, Inc.

Alluxio: Unify Data at Memory SpeedAlluxio, Inc.

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...Alluxio, Inc.

Accelerating Spark with KubernetesAlluxio, Inc.

AI/ML Infra Meetup | Maximizing GPU Efficiency : Optimizing Model Training wi...Alluxio, Inc.

Accelerating Spark Workloads in a Mesos Environment with AlluxioAlluxio, Inc.

More from Alluxio, Inc. (20)

PDF

Introduction to Apache Iceberg™ & TableflowAlluxio, Inc.

PDF

Optimizing Tiered Storage for Low-Latency Real-Time Analytics at AI ScaleAlluxio, Inc.

PDF

Meet in the Middle: Solving the Low-Latency Challenge for Agentic AIAlluxio, Inc.

PDF

From Data Preparation to Inference: How Alluxio Speeds Up AIAlluxio, Inc.

PDF

Best Practice for LLM Serving in the CloudAlluxio, Inc.

PDF

Meet You in the Middle: 1000x Performance for Parquet Queries on PB-Scale Dat...Alluxio, Inc.

PDF

How Coupang Leverages Distributed Cache to Accelerate ML Model TrainingAlluxio, Inc.

PDF

Alluxio Webinar | Inside Deepseek 3FS: A Deep Dive into AI-Optimized Distribu...Alluxio, Inc.

PDF

AI/ML Infra Meetup | Building Production Platform for Large-Scale Recommendat...Alluxio, Inc.

PDF

AI/ML Infra Meetup | How Uber Optimizes LLM Training and FinetuneAlluxio, Inc.

PDF

AI/ML Infra Meetup | Optimizing ML Data Access with Alluxio: Preprocessing, ...Alluxio, Inc.

PDF

AI/ML Infra Meetup | Deployment, Discovery and Serving of LLMs at Uber ScaleAlluxio, Inc.

PDF

Alluxio Webinar | What’s New in Alluxio AI: 3X Faster Checkpoint File Creatio...Alluxio, Inc.

PDF

AI/ML Infra Meetup | A Faster and More Cost Efficient LLM Inference StackAlluxio, Inc.

PDF

AI/ML Infra Meetup | Balancing Cost, Performance, and Scale - Running GPU/CPU...Alluxio, Inc.

PDF

AI/ML Infra Meetup | RAYvolution - The Last Mile: Mastering AI Deployment wit...Alluxio, Inc.

PDF

Alluxio Webinar | Accelerate AI: Alluxio 101Alluxio, Inc.

PDF

AI/ML Infra Meetup | The power of Ray in the era of LLM and multi-modality AIAlluxio, Inc.

PDF

AI/ML Infra Meetup | Exploring Distributed Caching for Faster GPU Training wi...Alluxio, Inc.

PDF

AI/ML Infra Meetup | Big Data and AI, Zoom DevelopersAlluxio, Inc.