Apache Spark Usage in the Open Source Ecosystem
2 likes2,550 views
The document discusses the usage statistics and library integration of Apache Spark among users of Databricks, highlighting user demographics and language preferences. Users have extensively adopted external libraries for ETL, visualization, and advanced analytics across Python, Scala, and R. The summary indicates a strong trend towards mixing languages and packages, showcasing the flexibility of Apache Spark in an open-source ecosystem.
Apache Spark Usage in the Open Source Ecosystem
- 1. Apache Spark Usage in the Open Source Ecosystem Hossein Falaki @mhfalaki
- 2. About me • Software Engineer /part-time Data Scientist atDatabricks • I started using Apache Spark since version 0.6 • Developed first version of Apache Spark CSV data source • Worked on SparkR and Rnotebooks at Databricks 2
- 3. Stackoverflow 2016 trending tech 3
- 4. Apache Spark Philosophy Unified engine Support end-to-end applications High-level APIs Easy to use, rich optimizations Integrate broadly Storage systems, libraries, etc SQLStreaming ML Graph … 1 2 3
- 5. Databricks Community Edition • In February Databricks launched a free version of its cloud based platform in beta • Since then more than 8,000 users registered • Users created over 61,000 notebooks indifferent languages • This is an analysis of third party libraries that our beta users imported to complement Apache Spark in Scala, Python, and R 5
- 6. What % of users use other libraries Language % users importing external libs Average # libs Median # libs Python 75 % 9 2 Scala 55 % 3 1 R 57 % 6 1 6
- 7. Installing libraries is easy 7
- 9. Most popular Python packages 9
- 11. What are these? ETL • re • datetime • pandas • json • csv • string • math /operator • urllib /urllib2 11 Visualization • matplotlib • ggplot • seaborn Advanced analytics • numpy • sklearn • graphframes • tensorflow • scipy Other • test_helper • os • md5
- 13. What packages go together? 13
- 15. Most popular Scala libraries 15
- 16. What are these? ETL • java/scala util • scala.collection • scala.math • java.{io, nio} • java.text • o.a.commons • kafka • twitter4j 16 Visualization • ? Advanced analytics • spark.ml • graphframes Other • java.net • scala.sys
- 18. What libraries go together? 18
- 19. R Packages 19
- 20. Most popular R packages 20
- 21. What are these? ETL • dplyr • plyr • reshape2 • jsonlite • tidyr • lubridate • httr • data.table 21 Visualization • ggplot2 • beanplot • plotly • ... Advanced analytics • sparkr • h2o • caret • e1071 Other • devtools • magrittr
- 23. Comparing Python, Scala & R 23
- 24. Languages have unique features 24 Scala/ Python / R R / Python Scala / Python/ R • 25 % of users,use multiple languages • 3% of notebooks mix different languages
- 25. Summary • Spark users extensively mix itwith other packages in different languages – One ofgoals ofSpark project is working well with other projects • ETL related libraries are the most popular category – Opportunities for newdata sources • Notebooks are being used for “small data” aswell as“big data.” • Languages and their ecosystems have diverse capabilities. Users seem to be mixing languages to their advantage – Scala is missing visualization libraries 25
- 26. Try your favorite library in Databricks 26 http://databricks.com/ce Try latest version of Apache Spark and previewof Spark 2.0
- 27. Thank you!
- 28. What packages are used together? 28