Keynote – From MapReduce to Spark: An Ecosystem Evolves by Doug Cutting, Chief Architect, Cloudera
4 likes1,043 views
The document summarizes the evolution from MapReduce to Apache Spark for data processing. Some key points: - MapReduce provided breakthroughs like data locality, fault tolerance, and scalability but the programming model required developing generally scalable solutions. - Apache Spark provides a richer, more expressive API that allows developing applications with 2-5x less code than MapReduce. It also provides fast in-memory execution up to an order of magnitude faster than MapReduce. - A survey found 82% of developers replaced MapReduce with Spark for its speed and ability to handle large datasets faster than MapReduce. Spark is now an important part of the Hadoop ecosystem.
Keynote – From MapReduce to Spark: An Ecosystem Evolves by Doug Cutting, Chief Architect, Cloudera
- 1. MapReduce to Apache Spark: An Ecosystem Evolves Doug Cutting (@cutting) Chief Architect & Co-founder of Apache Hadoop
- 2. Hadoop’s Original Architecture MapReduce Data Processing and Resource Management HDFS Filesystem/Storage
- 3. The MapReduce Breakthrough Key advances in MapReduce: • Data locality: Automatic split computation and appropriate launch of mappers • Fault-tolerance: Write-out of intermediate results and restartable mappers provides ability to run on commodity hardware • Linear scalability: Combination of locality + programming model forces developers to write generally scalable solutions Map Map Map Map Map Map Map Map Map Map Map Map Reduce Reduce Reduce Reduce
- 4. Apache Spark: A Better MapReduce Easy, Expressive API • Rich API (Java, Scala, and Python) • Interactive shell • 2-5x less code needed than MR Fast Execution • General execution graphs • In-memory storage • Order-of-magnitude improvement over MR
- 5. Big Data Developers are Rapidly Sparking Up Source: Typesafe Apache Spark Adoption Survey, Jan. 2015 • 82% have replaced MapReduce with Spark • 78% need faster processing for large data sets • 62% load data into Spark via HDFS • 22% of respondents run CDH, more than twice as many as any other Hadoop platform
- 6. Spark is now an important part of the Hadoop Platform
- 7. A Platform That Just Won’t Stop Growing NEWPROJECTS EXISTINGPROJECTS *CDHSUPPORTED Core Hadoop (HDFS, MapReduce) Solr Pig Core Hadoop HBase ZooKeeper Solr Pig Core Hadoop Hive Mahout HBase ZooKeeper Solr Pig Core Hadoop Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig Core Hadoop Flume Bigtop Oozie HCatalog Hue Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig YARN Core Hadoop Spark Tez Impala Kafka Drill Flume Bigtop Oozie HCatalog Hue Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig YARN Core Hadoop Parquet Sentry Spark Tez Impala Kafka Drill Flume Bigtop Oozie HCatalog Hue Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig YARN Core Hadoop Knox Flink Parquet Sentry Spark Tez Impala Kafka Drill Flume Bigtop Oozie HCatalog Hue Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig YARN Core Hadoop Kudu* RecordService* Ibis* Falcon Knox Flink Parquet* Sentry* Spark* Tez Impala* Kafka* Drill Flume* Bigtop* Oozie* Hcatalog* Hue* Sqoop* Avro* Hive* Mahout* Hbase* ZooKeeper* Solr* Pig* YARN* Core Hadoop* 2006 2008 2009 2010 2011 2012 20132007 2014 2015
- 8. Hadoop’s Next 10 Years Interest in public-cloud deployments are driving native support for them into the platform. Rapid hardware advances are forcing the community to re-think Hadoop’s foundations. Data sources are more numerous, distributed, and diverse (IoT), and Hadoop will adapt.