Hadoop at ayasdi
0 likesā¢752 views
The document provides an overview of Ayasdi's use of Hadoop, highlighting HDFS for storage, YARN for integration, and Parquet as the file format. It discusses the challenges and motivations behind using these technologies, the development of the BigDF framework for feature engineering, and its future direction. Additionally, it addresses the audience's background and experiences with big data tools during an interactive poll.
![bigdf - details
- open source since Sep 2014
- precedes Spark DataFrame, so built on spark-core
engine
- experimenting with Catalyst using Spark DataFrame
APIs, looks promising
- python and scala APIs
- feature engineering library [not open source :-( ]
- fast CSV reader(and other features) contributed to
spark-csv](https://image.slidesharecdn.com/hadoopatayasdi-150811021234-lva1-app6891/85/Hadoop-at-ayasdi-16-320.jpg)
Hadoop at ayasdi
- 1. Hadoop at Ayasdi Mohit Jaggi and Huang Xia, Zhen Li, Ajith Warrier
- 2. Overview - HDFS for storage - YARN for integration into Hadoop data lake - Parquet as the file format - bigdf based on Spark for feature engineering, data wrangling
- 3. Hadoop Data Lake bigdf Resource Broker Apache Spark YARN Algorithms UI/API handler Architecture
- 4. !! Audience Poll !! 1. How many data scientists? 2. How many backend engineers? 3. UI/frontend engineers? 4. Using hadoop in production? 5. Using Spark in production? 6. Personally worked on data bigger than 100GB? 1TB? 10TB? 1PB?
- 5. HDFS
- 6. HDFS - Motivation - installed base, large community - ecosystem to connect to most other data sources - commodity cluster - experiments with distributed NAS didn't show enough promise to justify the additional cost and complexity
- 7. HDFS - Usage - used as distributed storage - jobs dispatched to least loaded node - run Spark jobs
- 9. YARN - Motivation - Ayasdi scheduler - maximize throughput for batch jobs - minimize latency for interactive ātaskletsā - wanted to deploy in existing Hadoop data lakes - integrated inhouse scheduler with YARN - ātaskletsā get a long running container - batch jobs get a container on demand
- 10. YARN - Challenges - increased latency observable for small batch jobs - early adopter pains - sparse documentation - not the best API design
- 11. big data store compressed data
- 12. Parquet - Motivation - legacy: data stored in both row and column major - requires expensive transpose on ingestion - were designing a ātiled file formatā when discovered parquet
- 13. Parquet - Challenges - early adopter challenges - sparse documentation - needed to access package private APIs
- 14. big data
- 15. bigdf - Motivation - born out of experience using spark for feature engineering - creating classes for RDDs not reusable across projects - SQL not expressive enough
- 16. bigdf - details - open source since Sep 2014 - precedes Spark DataFrame, so built on spark-core engine - experimenting with Catalyst using Spark DataFrame APIs, looks promising - python and scala APIs - feature engineering library [not open source :-( ] - fast CSV reader(and other features) contributed to spark-csv
- 17. bigdf - future - wrapper around Spark DF - to protect from API changes - to add features e.g. āsparse column setā as āround- trip timeā for pull requests into large open source projects is high