Deep Learning Pipelines for High Energy Physics using Apache Spark with Distributed Keras on Analytics Zoo
1 like908 views
Luca Canali presents a deep learning pipeline at CERN for high energy physics data processing, utilizing Apache Spark and TensorFlow to address key challenges in event filtering for particle collisions at the LHC. The pipeline aims to improve the selection of interesting events while reducing false positives and operational costs. Key findings emphasize the importance of efficient data preparation and scalable training techniques in optimizing performance.
Deep Learning Pipelines for High Energy Physics using Apache Spark with Distributed Keras on Analytics Zoo
- 1. WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics
- 2. Luca Canali, CERN Deep Learning Pipelines for High Energy Physics using Apache Spark with Distributed Keras and Analytics Zoo #UnifiedDataAnalytics #SparkAISummit
- 3. About Luca 3#UnifiedDataAnalytics #SparkAISummit • Data Engineer at CERN – Hadoop and Spark service, database services – 19+ years of experience with data engineering • Sharing and community – Blog, notes, tools, contributions to Apache Spark @LucaCanaliDB – http://cern.ch/canali
- 4. CMS ALICE ATLAS LHCb CERN: Particle Accelerators (LHC) High Energy Physics Experiments
- 5. Experimental High Energy Physics is Data Intensive 5 Particle Collisions Physics Discoveries Large Scale Computing https://twiki.cern.ch/twiki/pub/CMSPublic/Hig13002TWiki/HZZ4l_animated.gif And https://iopscience.iop.org/article/10.1088/1742-6596/455/1/012027
- 6. Key Data Processing Challenge • Proton-proton collisions at LHC experiments happen at 40MHz. • Hundreds of TB/s of electrical signals that allow physicists to investigate particle collision events. • Storage, limited by bandwidth • Currently, only 1 every ~40K events stored to disk (~10 GB/s). 2018: 5 collisions/beam cross Current LHC 2026: 400 collisions/beam cross Future: High-Luminosity LHC upgrade
- 7. This can generate up to a petabyte of raw data per second Reduced to GB/s by filtering in real time Key is how to select potentially interesting events (trigger systems). PB/s 40 million collisions per second (Raw) 100,000 selections per second (L1) TB/s 1,000 selections per second (L2) GB/s Data Flow at LHC Experiments
- 8. R&D – Data Pipelines • Improve the quality of filtering systems • Reduce false positive rate • From rule-based algorithms to classifiers based on Deep Learning • Advanced analytics at the edge • Avoid wasting resources in offline computing • Reduction of operational costs
- 9. Particle Classifiers Using Neural Networks 1 63% 2 36% 3 1% Particle Classifier W + j QCD t-t̅ • R&D to improve the quality of filtering systems • Develop a “Deep Learning classifier” to be used by the filtering system • Goal: Identify events of interest for physics and reduce false positives • False positives have a cost, as wasted storage bandwidth and computing • “Topology classification with deep learning to improve real-time event selection at the LHC”, Nguyen et al. Comput.Softw.Big Sci. 3 (2019) no.1, 12
- 10. Deep Learning Pipeline for Physics Data Data Ingestion Feature Preparation Model Development Training Read physics data and feature engineering Prepare input for Deep Learning network 1. Specify model topology 2. Tune model topology on small dataset Train the best model Technology: the pipeline uses Apache Spark + Analytics Zoo and TensorFlow/Keras. Code on Python Notebooks.
- 11. Analytics Platform at CERN HEP software Experiments storage HDFS Personal storage Integrating new “Big Data” components with existing infrastructure: • Software distribution • Data platforms
- 13. Hadoop and Spark Clusters at CERN • Clusters: • YARN/Hadoop • Spark on Kubernetes • Hardware: Intel based servers, continuous refresh and capacity expansion Accelerator logging (part of LHC infrastructure) Hadoop - YARN - 30 nodes (Cores - 1200, Mem - 13 TB, Storage – 7.5 PB) General Purpose Hadoop - YARN, 65 nodes (Cores – 2.2k, Mem – 20 TB, Storage – 12.5 PB) Cloud containers Kubernetes on Openstack VMs, Cores - 250, Mem – 2 TB Storage: remote HDFS or EOS (for physics data)
- 14. Extending Spark to Read Physics Data • Physics data • Currently: >300 PBs of Physics data, increasing ~90 PB/year • Stored in the CERN EOS storage system in ROOT Format and accessible via XRootD protocol • Integration with Spark ecosystem • Hadoop-XRootD connector, HDFS compatible filesystem • Spark Datasource for ROOT format JNI Hadoop HDFS APIHadoop- XRootD Connector EOS Storage Service XRootD Client C++ Java https://github.com/cerndb/hadoop-xrootd https://github.com/diana-hep/spark-root
- 15. Labeled Data for Training and Test ● Simulated events ● Software simulators are used to generate events and calculate the detector response ● Raw data contains arrays of simulated particles and their properties, stored in ROOT format ● 54 million events
- 16. Step 1: Data Ingestion • Read input files: 4.5 TB from custom (ROOT) format • Feature engineering • Python and PySpark code, using Jupyter notebooks • Write output in Parquet format Output: • 25 M events • 950 GB in Parquet format • Target storage (HDFS) Input: • 54 M events ~4.5 TB • Physics data storage (EOS) • Physics data format (ROOT)
- 17. ● Filtering ● Multiple filters, keep only events of interest ● Example: “events with one electrons or muon with Pt > 23 Gev” • Prepare “Low Level Features” • Every event is associated to a matrix of particles and features (801x19) • High Level Features (HLF) • Additional 14 features are computed from low level particle features • Calculated based on domain-specific knowledge Feature Engineering
- 18. Step 2: Feature Preparation Features are converted to formats suitable for training • One Hot Encoding of categories • MinMax scaler for High Level Features • Sorting Low Level Features: prepare input for the sequence classifier, using a metric based on physics. This use a Python UDF. • Undersampling: use the same number of events for each of the three categories Result • 3.6 Million events, 317 GB • Shuffled and split into training and test datasets • Code: in a Jupyter notebook using PySpark with Spark SQL and ML
- 19. Performance and Lessons Learned • Data preparation is CPU bound • Heavy serialization-deserialization due to Python UDF • Ran using 400 cores: data ingestion took ~3 hours, • It can be optimized, but is it worth it ? • Use Spark SQL, Scala instead of Python UDF • Optimization: replacing parts of Python UDF code with Spark SQL and higher order functions: run time from 3 hours to 2 hours
- 20. Neural Network Models and 1. Fully connected feed-forward deep neural network • Trained using High Level Features (~1 GB of data) 2. Neural network based on Gated Recurrent Unit (GRU) • Trained using Low Level Features (~ 300 GB of data) 3. Inclusive classifier model • Combination of (1) + (2) Complexity + Classifier Performance
- 21. Hyper-Parameter Tuning– DNN • Hyper-parameter tuning of the DNN model • Trained with a subset of the data (cached in memory) • Parallelized with Spark, using spark_sklearn.grid_search • And scikit-learn + keras: tensorflow.keras.wrappers.scikit_learn
- 22. Deep Learning at Scale with Spark • Investigations and constraints for our exercise • How to run deep learning in a Spark data pipeline? • Neural network models written using Keras API • Deploy on Hadoop and/or Kubernetes clusters (CPU clusters) • Distributed deep learning • GRU-based model is complex • Slow to train on a single commodity (CPU) server
- 23. Spark, Analytics Zoo and BigDL • Apache Spark • Leading tool and API for data processing at scale • Analytics Zoo is a platform for unified analytics and AI • Runs on Apache Spark leveraging BigDL / Tensorflow • For service developers: integration with infrastructure (hardware, data access, operations) • For users: Keras APIs to run user models, integration with Spark data structures and pipelines • BigDL is an open source distributed deep learning framework for Apache Spark
- 24. BigDL Run as Standard Spark Programs Spark Program DL App on Driver Spark Executor (JVM) Spark Task BigDL lib Worker Intel MKL Standard Spark jobs Worker Worker Worker Worker Spark Executor (JVM) Spark Task BigDL lib Worker Intel MKL BigDL library Spark jobs BigDL Program Standard Spark jobs • No changes to the Spark or Hadoop clusters needed Iterative • Each iteration of the training runs as a Spark job Data parallel • Each Spark task runs the same model on a subset of the data (batch) Source: Intel BigDL Team
- 25. BigDL Parameter Synchronization Source: https://github.com/intel-analytics/BigDL/blob/master/docs/docs/whitepaper.md
- 26. Model Development – DNN for HLF • Model is instantiated using the Keras- compatible API provided by Analytics Zoo
- 27. Model Development – GRU + HLF A more complex network topology, combining a GRU of Low Level Feature + a DNN of High Level Features
- 28. Distributed Training Instantiate the estimator using Analytics Zoo / BigDL The actual training is distributed to Spark executors Storing the model for later use
- 29. Analytics Zoo/BigDL on Spark scales up in the ranges tested Performance and Scalability of Analytics Zoo/BigDL Inclusive classifier model DNN model, HLF features
- 30. Workload Characterization • Training with Analytics zoo • GRU-based model: Distributed training on YARN cluster • Measure with Spark Dashboard: it is CPU bound
- 31. Results – Model Performance • Trained models with Analytics Zoo and BigDL • Met the expected results for model performance: ROC curve and AUC
- 32. Training with TensorFlow 2.0 • Training and test data • Converted from Parquet to TFRecord format using Spark • TensorFlow: data ingestion using tf.data and tf.io • Distributed training with tf.distribute + tool for K8S: https://github.com/cerndb/tf-spawner Distributed training with TensorFlow 2.0 on Kubernetes (CERN cloud) Distributed training of the Keras model with: tf.distribute.experimental. MultiWorkerMirroredStrategy
- 33. Performance and Lessons Learned • Measured distributed training elapsed time • From a few hours to 11 hours, depending on model, number of epochs and batch size. Hard to compare different methods and solutions (many parameters) • Distributed training with BigDL and Analytics Zoo • Integrates very well with Spark • Need to cache data in memory • Noisy clusters with stragglers can add latency to parameter synchronization • TensorFlow 2.0 • It is straightforward to distribute training on CPUs and GPUs with tf.distribute • Data flow: Use TFRecord format, read with TensorFlow’s tf.data and tf.io • GRU training performance on GPU: 10x speedup in TF 2.0 • Training of the Inclusive Classifier on a single P100 in 5 hours
- 34. Data and models from Research Input: labeled data and DL models Feature engineering at scale Distributed model training Output: particle selector model Hyperparameter optimization (Random/Grid search) Recap: our Deep Learning Pipeline with Spark
- 35. Model Serving and Future Work • Using Apache Kafka and Spark? • FPGA serving DNN models MODEL Streaming platform MODEL RTL translation FPGA Output pipeline: to storage / further online analysis Output pipeline
- 36. Summary • The use case developed addresses the needs for higher efficiency in event filtering at LHC experiments • Spark, Python notebooks • Provide well-known APIs and productive environment for data preparation • Data preparation performance, lessons learned: • Use Spark SQL/DataFrame API, avoid Python UDF when possible • Successfully scaled Deep Learning on Spark clusters • Using Analytics Zoo and BigDL • Deployed on existing Intel Xeon-based servers: Hadoop clusters and cloud • Good results also with Tensorflow 2.0, running on Kubernetes • Continuous evolution and improvements of DL at scale • Data preparation and scalable distributed training are key
- 37. Acknowledgments • Matteo Migliorini, Marco Zanetti, Riccardo Castellotti, Michał Bień, Viktor Khristenko, CERN Spark and Hadoop service, CERN openlab • Authors of “Topology classification with deep learning to improve real-time event selection at the LHC”, notably Thong Nguyen, Maurizio Pierini • Intel team for BigDL and Analytics Zoo: Jiao (Jennie) Wang, Sajan Govindan – Analytics Zoo: https://github.com/intel-analytics/analytics-zoo – BigDL: https://software.intel.com/bigdl References: – Data and code: https://github.com/cerndb/SparkDLTrigger – Machine Learning Pipelines with Modern Big Data Tools for High Energy Physics http://arxiv.org/abs/1909.10389 #UnifiedDataAnalytics #SparkAISummit 37
- 38. DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT