Sparkling pandas Letting Pandas Roam - PyData Seattle 2015
16 likes•4,606 views
The document discusses 'Sparkling Pandas,' a tool for extending the functionality of Pandas by leveraging Apache Spark for distributed data processing. It outlines the key differences between Pandas and Spark, providing examples of how to convert data between the two frameworks and extend Spark's capabilities with custom functions. The document also includes references to resources for further learning and the community support for Sparkling Pandas.
![Convert a Spark DataFrame to Pandas
import pandas
...
ddf = sqlContext.read.json("hdfs://...")
# Some Spark transformations
transformedDdf = ddf.filter(ddf['age'] > 21)
return transformedDdf.toPandas()](https://image.slidesharecdn.com/sparklingpandas-electricbugaloo-pydataseattle20152-150725192543-lva1-app6891/85/Sparkling-pandas-Letting-Pandas-Roam-PyData-Seattle-2015-11-320.jpg)
![Simple graphing with Sparkling Pandas
import matplotlib.pyplot as plt
plot = speaker_pronouns["pronoun"].plot()
plot.get_figure().savefig("/tmp/fig")
Not yet
merged in](https://image.slidesharecdn.com/sparklingpandas-electricbugaloo-pydataseattle20152-150725192543-lva1-app6891/85/Sparkling-pandas-Letting-Pandas-Roam-PyData-Seattle-2015-20-320.jpg)
Sparkling pandas Letting Pandas Roam - PyData Seattle 2015
- 1. Sparkling Pandas Scaling Pandas beyond a single machine (or letting Pandas Roam) With Special thanks to Juliet Hougland :)
- 2. Sparkling Pandas Scaling Pandas beyond a single machine (or letting Pandas Roam) With Special thanks to Juliet Hougland :)
- 3. Who am I? Holden â—Ź I prefer she/her for pronouns â—Ź Co-author of the Learning Spark book â—Ź Engineer at Alpine Data Labs â—‹ previously DataBricks, Google, Foursquare, Amazon â—Ź @holdenkarau â—Ź http://www.slideshare.net/hkarau â—Ź https://www.linkedin.com/in/holdenkarau
- 4. What is Pandas? user_id panda_ty pe 01234 giant 12345 red 23456 giant 34567 giant 45678 red 56789 giant â—Ź DataFrames--Indexed, tabular data structures â—Ź Easy slicing, indexing, subsetting/filtering â—Ź Excellent support for time series data â—Ź Data alignment and reshaping http://pandas.pydata.org/
- 5. What is Spark? Fast general engine for in memory data processing. tl;dr - 100x faster than Hadoop MapReduce*
- 6. The different pieces of Spark Apache Spark SQL & DataFrames Streaming Language APIs Scala, Java, Python, & R Graph Tools Spark ML bagel & Grah X MLLib Community Packages
- 7. Some Spark terms Spark Context (aka sc) â—Ź The window to the world of Spark sqlContext â—Ź The window to the world of DataFrames Transformation â—Ź Takes an RDD (or DataFrame) and returns a new RDD or DataFrame Action â—Ź Causes an RDD to be evaluated (often storing the result)
- 8. Dataframes between Spark & Pandas Spark â—Ź Fast â—Ź Distributed â—Ź Limited API â—Ź Some ML â—Ź I/O Options â—Ź Not indexed Pandas â—Ź Fast â—Ź Single Machine â—Ź Full Feature API â—Ź Integration with ML â—Ź Different I/O Options â—Ź Indexed â—Ź Easy to visualize
- 9. Panda IMG by Peter Beardsley
- 10. Simple Spark SQL Example input = sqlContext.jsonFile(inputFile) input.registerTempTable("tweets") topTweets = sqlContext.sql("SELECT text, retweetCount" + "FROM tweets ORDER BY retweetCount LIMIT 10") local = topTweets.collect()
- 11. Convert a Spark DataFrame to Pandas import pandas ... ddf = sqlContext.read.json("hdfs://...") # Some Spark transformations transformedDdf = ddf.filter(ddf['age'] > 21) return transformedDdf.toPandas()
- 12. Convert a Pandas DataFrame to Spark import pandas ... df = panda.DataFrame(...) ... ddf = sqlContext.DataFrame(df)
- 13. Let’s combine the two ● Spark DataFrames already provides some of what we need ○ Add UDFs / UDAFS ○ Use bits of Pandas code ● http://spark-packages.org - excellent pace to get libraries
- 14. So where does the PB&J go? Spark DataFrame Sparkling Pandas API Custom UDFS Pandas Code Sparkling Pandas Scala Code PySpark RDDs Pandas Code Internal State
- 15. Extending Spark - adding index support self._index_names def collect(self): """Collect the elements in an Dataframe and concatenate the partition.""" df = self._schema_rdd.toPandas() df = _update_index_on_df(df, self._index_names) return df
- 16. Extending Spark - adding index support def _update_index_on_df(df, index_names): if index_names: df = df.set_index(index_names) # Remove names from unnamed indexes index_names = _denormalize_names(index_names) df.index.names = index_names return df
- 17. Adding a UDF in Python sqlContext.registerFunction("strLenPython", lambda x: len(x), IntegerType())
- 18. Extending Spark SQL w/Scala for fun & profit // functions we want to be callable from python object functions { def kurtosis(e: Column): Column = new Column(Kurtosis(EvilSqlTools.getExpr(e))) def registerUdfs(sqlCtx: SQLContext): Unit = { sqlCtx.udf.register("rowKurtosis", helpers.rowKurtosis _) } }
- 19. Extending Spark SQL w/Scala for fun & profit def _create_function(name, doc=""): def _(col): sc = SparkContext._active_spark_context f = sc._jvm.com.sparklingpandas.functions, name jc = getattr(f)(col._jc if isinstance(col, Column) else col) return Column(jc) return _ _functions = { 'kurtosis': 'Calculate the kurtosis, maybe!', }
- 20. Simple graphing with Sparkling Pandas import matplotlib.pyplot as plt plot = speaker_pronouns["pronoun"].plot() plot.get_figure().savefig("/tmp/fig") Not yet merged in
- 21. Why is SparklingPandas fast*? Keep stuff in the JVM as much as possible. Lazy operations Distributed *For really flexible versions of the word fast Coffee by eltpics Panda image by Stéfan Panda image by cactusroot
- 22. Supported operations: DataFrames â—Ź to_spark_sql â—Ź applymap â—Ź groupby â—Ź collect â—Ź stats â—Ź query â—Ź axes â—Ź ftype â—Ź dtype Context â—Ź simple â—Ź read_csv â—Ź from_data_frame â—Ź parquetFile â—Ź read_json â—Ź stop GroupBy â—Ź groups â—Ź indices â—Ź first â—Ź median â—Ź mean â—Ź sum â—Ź aggregate
- 23. Always onwards and upwards Now Hypothetical, Wonderful Future Workdone Time
- 24. Related Works Blaze ● http://continuum.io/blog/blaze AdaTao’s Distributed DataFrame ● http://spark-summit.org/2014/talk/distributed-dataframe- ddf-on-apache-spark-simplifying-big-data-for-the-rest-of- us Numba ● http://numba.pydata.org/
- 25. Using Sparkling Pandas You can get Sparkling Pandas from â—Ź Website: http://www.sparklingpandas.com â—Ź Code: https://github.com/sparklingpandas/sparklingpandas â—Ź Mailing List https://groups.google.com/d/forum/sparklingpandas
- 26. Getting Sparkling Pandas friends The examples from this will get merged into master. Pandas â—Ź http://pandas.pydata.org/ (or pip) Spark â—Ź http://spark.apache.org/
- 27. many pandas by David Goehring Any questions?