VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger Menezes
9 likes4,357 views
Vegas is a visualization library for Scala, designed to fill the gap in the Scala/Spark ecosystem similar to Matplotlib in Python. It allows for declarative statistical visualizations with built-in operations like filtering and aggregation, enhancing productivity in machine learning experimentation. The library also supports various plot types, interactive selections, and is designed for seamless integration within notebooks.
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger Menezes
- 1. Vegas The Missing Matplotlib for Scala/Spark DB Tsai Roger Menezes
- 2. Homepage Kids Page Downloads Page Netflix Recommendations Every aspect of the Experience is Machine Learned
- 3. 3 2017 > 100M members > 190 countries
- 5. Genres: 23 rows/page average Sims: 10 rows/page average
- 6. My List: Continue Watching: Popular on Netflix: Trending Now: Watch It Again: Top Picks: Because You Watched: Genres: New Releases: Recently Added: Originals RowBillboard:
- 7. Machine Learning at Netflix ● Optimize the Experimentation usecase vs Productionization ● Experimentation ○ Opportunity sizing, Data Exploration ○ Feature Identification and Selection ○ Tweaks to ML algos ○ Model Evaluation
- 9. Notebooks ● Optimal for Experimentation ● Sharing reproducible research ○ Facilitates feedback loop with Product Managers ● End to end ML experiment. ○ Interactivity drives productivity
- 10. Python Notebooks
- 11. Python Notebooks ● Seamless Experience - ML experimentation ● Well known Scientific computing libraries ● Huge catalog of Visualization plotting libraries ○ Matplotlib, Seaborn, Bokeh, BQPlot, Lightning, etc.
- 12. Scala Notebooks ● Zeppelin, Jupyter, Databricks, Spark-Notebooks, ... ● Computing library gap filling up ● Lack of Visualization Libraries ○ Main friction point in adoption ○ End to End ML use case not convincing
- 13. Introducing Vegas ● Visualization Library in Scala ● Mainly built for the notebook use case ● Scala wrapper around Vega-Lite ○ Missing MatPlotLib for the Scala/Spark world.
- 14. DECLARATIVE STATISTICAL VISUALIZATION GRAMMAR IN SCALA You tell it WHAT should be done with the data, and it knows HOW to do it! Operations such as filtering, aggregation, faceting are built into the visualization, rather than putting the burden on the user to massage the data into shape. Complex visualizations can be built with a few high level abstractions: DATA TRANS- FORMS SCALES GUIDES MARKS cf : Altair Talk by Brian Granger in PyData 2016 https://youtu.be/v5mrwq7yJc4
- 15. Added Bonus of Declarative Visualizations: INTERACTIVITY! D3JS VEGAS VEGAS CODE EXPANDS OUT TO D3JS CODE!
- 16. Anatomy of a plot: Channels X/Y channel Shape Channel Size Channel Color Channel
- 18. Features…
- 19. 1. Supports most plot types
- 20. 2. Trellis plots
- 21. 3. Layers Layer 1. Layer 2. Layer 3.
- 22. 4. Notebook and Consoles
- 23. 5. Built-in spark support Vegas .withDataFrame(myDataFrame) .encodeX(“population”) .encodeY(“age”) Mapped Columns Pass In DF.
- 24. 6. Visual statistics ● Advanced Binning ● Sorting ● Scaling ● Custom Transforms ● Time Series ● Aggregation ● Filtering ● Math functions (log, etc) ● Descriptive Statistics
- 25. How It Works !
- 26. 1. Specify in Scala 2. Embed HTML (iFrame) 3. Render within iFrame using JS
- 27. VEGA D3JS VEGA-LITE* VEGAS MOREABSTRACTION SCALA DSL EMITS TYPE-CHECKED VEGA-LITE JSON VEGA-LITE CONVERTS INTERNALLY TO VEGA JSON SPEC VEGA TRANSLATES JSON TO D3JS CODE THAT CAN BE VERY VERBOSE A SCALA DSL FOR VEGA-LITE * Vega-Lite
- 29. What’s coming
- 32. Contributors Aish DB Roger Sudeep Jeremy
- 33. Thank you.
- 34. @NetflixResearch @rogermenezes @dbtsai The missing MatPlotLib for Scala/Spark http://vegas-viz.org