Rethinking data intensive science using scalable analytics systems
1 like371 views
This paper describes ADAM, a genomics pipeline that uses Apache Spark and Parquet to achieve a 28x speedup over current pipelines while reducing costs by 63%. The paper discusses how ADAM leverages techniques like columnar storage, Spark's distributed processing, and data locality to improve performance. Evaluation shows ADAM outperforms tools like GATK and Sambamba on tasks like variant calling and duplicate marking. The system achieves near-linear scaling to 128 nodes, enabling faster and cheaper genomic analysis through distributed processing on commodity clusters.
![• We evaluated ADAM against the GATK [14], SAMtools
[32], Picard [51], and Sambamba [50]. We evaluated the
performance of BQSR, INDEL realignment (IR), duplicate
marking (DM), sort, and Flagstat (FS).
Genomics Workloads
12](https://image.slidesharecdn.com/rethinkingdata-intensivescienceusingscalableanalyticssystems-150915153343-lva1-app6891/85/Rethinking-data-intensive-science-using-scalable-analytics-systems-12-320.jpg)
![• This data is shown in Table 2. Although ADAM is more
expensive than the best legacy tool (Sambamba [50]) for
sorting and duplicate marking, ADAM is less expensive
for all other stages. In total, using ADAM reduces the
end-to-end analysis cost by 63% over a pipeline
constructed out of solely legacy tools.
Genomics Workloads
13](https://image.slidesharecdn.com/rethinkingdata-intensivescienceusingscalableanalyticssystems-150915153343-lva1-app6891/85/Rethinking-data-intensive-science-using-scalable-analytics-systems-13-320.jpg)
Rethinking data intensive science using scalable analytics systems
- 1. Rethinking data-intensive science using scalable analytics systems Frank Austin Nothaft, Matt Massie, Timothy Danford, Zhao Zhang, Uri Laserson, Carl Yeksigian, Jey Kottalam, Arun Ahuja, Jeff Hammerbacher, Michael Linderman, Michael J. Franklin, Anthony D. Joseph, David A. Patterson AMPLab, University of California, Berkeley, Cloudera, San Francisco, CA, Carl Icahn School of Medicine, Mount Sinai, New York, NY, Genomebridge, Cambridge, MA 1
- 2. Abstract • In this paper, we describe ADAM, an example genomics pipeline that leverages the open-source Apache Spark and Parquet systems to achieve a 28x speedup over current genomics pipelines, while reducing cost by 63%. From building this system, we were able to distill a set of techniques for implementing scientific analyses efficiently using commodity “big data” systems. 2
- 3. Background source : NIH National Genome Research Institute 3
- 4. Characteristics of science analysis systems 4
- 5. Layering • Physical Storage coordinates data writes to physical media. • Data Distribution manages access, replication, and distribution of the files that have been written to storage media. • Materialized Data encodes the patterns for how data is encoded and stored. This layer determines I/O bandwidth and compression. 5
- 6. Layering • Data Schema specifies the representation of data, and forms the narrow waist of the stack that separates access from execution • Evidence Access provides primitives for processing data, and enables the transformation of data into different views and traversals. • Presentation enhances the data schema with convenience methods for performing common tasks and accessing common derived fields from a single element. 6
- 7. Layering • Applications use the evidence access and presentation layers to compose algorithms for performing an analysis. 7
- 9. Parquet • OSS Created by Twitter and Cloudera, based on Google Dremel • Columnar File Format • Limit I/O to only data that is needed • Compresses very well - ADAM file are 5-25% smaller than BAM file without loss of data • 3 layers of parallelism: File/row group, Column chunk, Page 9
- 10. Parquet/Spark integration • 1 row group in Parquet maps to 1 partition in spark • We interact with Parquet via input/output formats • Spark builds and execute a computation Directed Acyclic Graph(DAG), manages data locality, error/retries 10
- 11. Performance 11
- 12. • We evaluated ADAM against the GATK [14], SAMtools [32], Picard [51], and Sambamba [50]. We evaluated the performance of BQSR, INDEL realignment (IR), duplicate marking (DM), sort, and Flagstat (FS). Genomics Workloads 12
- 13. • This data is shown in Table 2. Although ADAM is more expensive than the best legacy tool (Sambamba [50]) for sorting and duplicate marking, ADAM is less expensive for all other stages. In total, using ADAM reduces the end-to-end analysis cost by 63% over a pipeline constructed out of solely legacy tools. Genomics Workloads 13
- 14. • Table 3 describes the instance types. Genomics Workloads 14
- 15. Genomics Workloads • We achieve near-linear speedup across 128 nodes 15
- 16. Conclusion • By rethinking the architecture of scientific data management systems, we have been able to achieve parity on single node systems, while providing linear strong scaling out to 128 nodes. By making it easy to scale scientific analysis across multiple commodity machines, we enable the use of smaller, less expensive computers, leading to a 63% cost improvement and a 28x improvement in read preprocessing pipeline latency. 16
- 17. Q&A 17