Austin Data Meetup 092014 - Spark
1 like•669 views
The Spark Summit was attended by over 1,100 people from 450+ companies and featured keynotes and community presentations. Spark is an active Apache project with over 250 contributors from 50+ companies. It includes subprojects for SQL, streaming, machine learning and graph processing. Popular use cases include real-time recommendations, cancer genomics, and media analytics.
Austin Data Meetup 092014 - Spark
- 1. Spark
- 2. Spark - Summit - News - Basics - Advanced - Subprojects - Use Cases - Resources
- 3. Summit - 1,164 participants from over 453 companies attended - Spark Training sold out at 300 participants - 31 organizations sponsored the event - 12 keynotes and 52 community presentations were given
- 4. News - Project - Databricks
- 5. Project - 1.0.0 release - Graduated incubator - Very active community
- 6. Very active community - Top three Apache projects - Most active Big Data project - > 50 companies - > 250 contributors - > 175,000 LOC
- 7. Databricks - Certification - Cloud
- 8. Certification - Every certified app will run on every certified distribution - Distribution Partners - App Partners
- 9. Distribution Partners - Cloudera - MapR - Hortonworks - Pivotal - IBM - Amazon Web Services - SAP
- 10. App Partners - Alteryx - Datastax - 0xdata - Typesafe - Zoomdata
- 11. Cloud - Vision: Make Big Data Easy! - Product: Badass - Hosted Platform - Cluster Management - Interactive Workspace
- 12. Interactive Workspace - Notebooks - Dashboards - Jobs
- 13. Dashboards - WYSIWYG Builder - Interactive plots - One-click publishing
- 14. Spark Basics - Execution - RDDs - Caching - Broadcast - Languages
- 15. Execution - Apply Functional Operators across Distributed Collections - Master / Worker - Lazy - Parallelize with Threads first
- 16. RDDs - Interface for dataset - Backed by anything - Any InputFormat class - HDFS default
- 17. Caching - Store intermediate results in memory - Partition-locality - Significant speed-up for iterative algorithms
- 18. Broadcast - Send immutable object to all workers - Similar to DistributedCache in mapreduce
- 19. Languages - Scala - Python - Java 7 - Java 8 - R - Clojure
- 20. Advanced - Partitioning - Persistence Options - Checkpointing - Accumulators - Optimizations
- 21. Subprojects - SparkSQL - Tachyon - Spark Streaming - MLLib - GraphX - BlinkDB - Spark Job Server
- 22. SparkSQL - Replaces Shark - Core - Catalyst - Libraries
- 23. Core - SchemaRDDs - Query Execution - Caching
- 24. Catalyst - Relational algebra - Expressions / UDFs - Query Planning - Optimizer
- 25. Libraries - POJOs - JDBC - JSON - Parquet - Hive
- 26. Hive - Catalog info from Metastore - Helps connect UI like Microstrategy / Tableau - Wrappers for UDF, UDAFs, UDTFs - Supports TRANSFORM - Supports SerDes
- 27. Tachyon - In Memory (Off-Heap) Distributed Datastore - Change URI from hdfs:// to tachyon:// - Share datasets between jobs without HDFS - Helps scaling by off-loading allocation responsibility and GC pauses from executor processes
- 28. Spark Streaming - Real-time streams - Micro-batching - Windowed Computations - Lambda Architecture
- 29. MLLib - Summary statistics - Regression - Classification - Clustering - Collaborative Filtering - Optimization - Dimensional Reduction
- 30. GraphX - Graph, VertexRDD, EdgeRDD objects and operations - Pregel API - mapReduceTriplets List<V,E,V> - Graph analytics libraries
- 31. Graph analytics libraries - ConnectedComponents - PageRank - TriangleCount - ShortestPaths - SVDPlusPlus
- 32. BlinkDB - Get estimated results - Time bound - Error bound
- 33. Spark Job Server - Runs multiple jobs / contexts in same process - Allows for RDD Caching / Sharing between jobs - Job Persistence
- 34. Use Cases - Spotify - Real-time Auctions - ShareThrough - Real-time Recommendations - Graphflow - Cancer Genomics - AMPLab - Malware Detection - F-Secure - Media Distribution Analytics - NBC Universal - Personal Fitness - Jawbone - Neuroscience - HHMI
- 35. Resources - Code - Event - Technology - Videos
- 37. Event - spark-summit.org - http://arjon.es/2014/06/30/spark-summit-2014-day-1/ - https://www.crowdchat.net/chat/c3BvdF9vYmpfODc=. - https://nathanbrixius.wordpress.com/2014/07/02/spark-summit-keynote- notes/ - http://thomaswdinsmore.com/2014/07/03/spark-summit-2014- roundup/
- 38. Technology - Learning Spark (O'Reilly eBook) - www.spark-stack.org - ampcamp.berkeley.edu - https://amplab.cs.berkeley.edu/2013/10/23/got-a-minute-spin- up-a-spark-cluster-on-your-laptop-with-docker/
- 39. YouTube - AmpLab https://www.youtube.com/channel/UCWudC4d9i-2yxR5tuen- Nuw - Databricks https://www.youtube.com/channel/UC3q8O3Bh2Le8Rj1-Q- _UUbA - Apache Spark https://www.youtube.com/channel/UCRzsq7k4-kT-h3TDUBQ82- w