GraphFrames: DataFrame-based graphs for Apache® Spark™
31 likes7,368 views
GraphFrames is a library built on Apache Spark that simplifies graph processing by leveraging DataFrames for higher-level APIs and optimizations. It offers capabilities like motif finding, graph algorithms, and integrates seamlessly with Spark's existing functionalities while allowing for the use of Java, Python, and Scala. The document highlights its architecture, use cases, and a roadmap for future enhancements and community contributions.
![Motif finding
24
IAD
JFK
LAX
SFO
SEA
DFW
Search for structural
patterns within a graph.
val paths: DataFrame =
g.find(“(a)-[e1]->(b);
(b)-[e2]->(c);
!(c)-[]->(a)”)](https://image.slidesharecdn.com/graphframesdataframe-basedgraphsforapachespark-160414180534/85/GraphFrames-DataFrame-based-graphs-for-Apache-Spark-24-320.jpg)
![Motif finding
25
IAD
JFK
LAX
SFO
SEA
DFW
(b)
(a)Search for structural
patterns within a graph.
val paths: DataFrame =
g.find(“(a)-[e1]->(b);
(b)-[e2]->(c);
!(c)-[]->(a)”)](https://image.slidesharecdn.com/graphframesdataframe-basedgraphsforapachespark-160414180534/85/GraphFrames-DataFrame-based-graphs-for-Apache-Spark-25-320.jpg)
![Motif finding
26
IAD
JFK
LAX
SFO
SEA
DFW
(b)
(a)
(c)
Search for structural
patterns within a graph.
val paths: DataFrame =
g.find(“(a)-[e1]->(b);
(b)-[e2]->(c);
!(c)-[]->(a)”)](https://image.slidesharecdn.com/graphframesdataframe-basedgraphsforapachespark-160414180534/85/GraphFrames-DataFrame-based-graphs-for-Apache-Spark-26-320.jpg)
![Motif finding
27
IAD
JFK
LAX
SFO
SEA
DFW
(b)
(a)
(c)
Search for structural
patterns within a graph.
val paths: DataFrame =
g.find(“(a)-[e1]->(b);
(b)-[e2]->(c);
!(c)-[]->(a)”)](https://image.slidesharecdn.com/graphframesdataframe-basedgraphsforapachespark-160414180534/85/GraphFrames-DataFrame-based-graphs-for-Apache-Spark-27-320.jpg)
![Motif finding
28
IAD
JFK
LAX
SFO
SEA
DFW
Search for structural
patterns within a graph.
val paths: DataFrame =
g.find(“(a)-[e1]->(b);
(b)-[e2]->(c);
!(c)-[]->(a)”)
(b)
(a)
(c)
Then filter using vertex &
edge data.
paths.filter(“e1.delay > 20”)](https://image.slidesharecdn.com/graphframesdataframe-basedgraphsforapachespark-160414180534/85/GraphFrames-DataFrame-based-graphs-for-Apache-Spark-28-320.jpg)
![GraphFrames vs. GraphX
38
GraphFrames GraphX
Builton DataFrames RDDs
Languages Scala, Java, Python Scala
Use cases Queries & algorithms Algorithms
Vertex IDs Any type (in Catalyst) Long
Vertex/edg
e attributes
Any number of
DataFrame columns
Any type (VD, ED)
Return
types
GraphFrame or
DataFrame
Graph[VD, ED], or
RDD[Long, VD]](https://image.slidesharecdn.com/graphframesdataframe-basedgraphsforapachespark-160414180534/85/GraphFrames-DataFrame-based-graphs-for-Apache-Spark-38-320.jpg)
![GraphX compatibility
Simple conversionsbetweenGraphFrames& GraphX.
val g: GraphFrame = ...
// Convert GraphFrame à GraphX
val gx: Graph[Row, Row] = g.toGraphX
// Convert GraphX à GraphFrame
val g2: GraphFrame = GraphFrame.fromGraphX(gx)
39
Vertex & edgeattributes
are Rows in order to
handlenon-LongIDs
Wrapping existing GraphX code: See Belief Propagation example:
https://github.com/graphframes/graphframes/blob/master/src/main/scala/org/graphframes/examples/BeliefPropagation.scala](https://image.slidesharecdn.com/graphframesdataframe-basedgraphsforapachespark-160414180534/85/GraphFrames-DataFrame-based-graphs-for-Apache-Spark-39-320.jpg)
GraphFrames: DataFrame-based graphs for Apache® Spark™
- 1. GraphFrames DataFrame-based graphs for Apache® Spark™ Joseph K. Bradley 4/14/2016
- 2. About the speaker: Joseph Bradley Joseph Bradley is a Software Engineerand Apache Spark PMC member working on MLlib at Databricks. Previously,he was a postdoc at UC Berkeley after receiving hisPh.D. in Machine Learning from Carnegie Mellon U.in 2013.His research included probabilistic graphical models, parallel sparse regression, and aggregation mechanismsfor peergrading in MOOCs. 2
- 3. About the moderator: Denny Lee Denny Lee is a Technology Evangelistwith Databricks; he is a hands-on data sciencesengineer with more than 15 years of experience developing internet-scale infrastructure, data platforms, and distributed systems for both on-premisesand cloud. Prior to joining Databricks, Denny worked as a SeniorDirector of Data SciencesEngineering at Concur and was part of the incubation teamthat builtHadoop on Windowsand Azure (currently known as HDInsight). 3
- 4. We are Databricks, the company behind Apache Spark Founded by the creators of Apache Spark in 2013 Share of Spark code contributed by Databricks in 2014 75% 4 Data Value Created Databricks on top of Spark to make big data simple.
- 5. … Apache Spark Engine Spark Core Spark Streaming Spark SQL MLlib GraphX Unified engineacross diverse workloads & environments Scale out, fault tolerant Python, Java, Scala, and R APIs Standard libraries
- 7. NOTABL E USERS THAT PRESENTED AT SPARK SUMMIT 2 0 1 5 SAN F RANCISCO Source: Slide5ofSparkCommunityUpdate
- 8. Outline GraphFrames overview GraphFrames vs. GraphXand other libraries Details for power users Roadmap and resources 8
- 9. Outline GraphFrames overview GraphFrames vs. GraphXand other libraries Details for power users Roadmap and resources 9
- 10. Graphs 10 vertex edge id City State “JFK” “New York” NY Example: airports & flights between them JFK IAD LAX SFO SEA DFW src dst delay tripID “JFK ” “SEA” 45 1058923
- 11. Apache Spark’s GraphX library Overview • General-purpose graph processinglibrary • Optimized for fast distributedcomputing • Library of algorithms: PageRank, Connected Components,etc. 11 Challenges • No Java, PythonAPIs • Lower-levelRDD-based API (vs.DataFrames) • Cannot use recent Spark optimizations:Catalyst query optimizer,Tungsten memory management
- 12. Enter GraphFrames Goal: DataFrame-based graphson ApacheSpark • Simplify interactive queries • Support motif-findingforstructural pattern search • Benefitfrom DataFrame optimizations Collaboration between Databricks, UC Berkeley& MIT + Now with community contributors! 12
- 13. Graphs 13 vertex edge id City State “JFK” “New York” NY Example: airports & flights between them JFK IAD LAX SFO SEA DFW src dst delay tripID “JFK ” “SEA” 45 1058923
- 14. GraphFrames “vertices” DataFrame • 1 vertexper Row • id: column with unique ID “edges” DataFrame • 1 edge per Row • src, dst: columns using IDs from vertices.id 14 Extra columns store vertexor edge data (a.k.a. attributes or properties). id City State “JFK” “New York” NY “SEA” “Seattle” WA src dst delay tripID “JFK” “SEA” 45 1058923 “DFW” “SFO” -7 4100224
- 16. 16
- 19. Queries Simple queries Motif finding Graph algorithms 19
- 20. Simple queries SQL queries on vertices & edges E.g., what trips are most likely to have significantdelays? 20 Graph queries • Vertex degrees • # edgesper vertex(incoming,outgoing,total) • Triplets • Join vertices and edgesto get (src, edge,dst)
- 21. 21
- 24. Motif finding 24 IAD JFK LAX SFO SEA DFW Search for structural patterns within a graph. val paths: DataFrame = g.find(“(a)-[e1]->(b); (b)-[e2]->(c); !(c)-[]->(a)”)
- 25. Motif finding 25 IAD JFK LAX SFO SEA DFW (b) (a)Search for structural patterns within a graph. val paths: DataFrame = g.find(“(a)-[e1]->(b); (b)-[e2]->(c); !(c)-[]->(a)”)
- 26. Motif finding 26 IAD JFK LAX SFO SEA DFW (b) (a) (c) Search for structural patterns within a graph. val paths: DataFrame = g.find(“(a)-[e1]->(b); (b)-[e2]->(c); !(c)-[]->(a)”)
- 27. Motif finding 27 IAD JFK LAX SFO SEA DFW (b) (a) (c) Search for structural patterns within a graph. val paths: DataFrame = g.find(“(a)-[e1]->(b); (b)-[e2]->(c); !(c)-[]->(a)”)
- 28. Motif finding 28 IAD JFK LAX SFO SEA DFW Search for structural patterns within a graph. val paths: DataFrame = g.find(“(a)-[e1]->(b); (b)-[e2]->(c); !(c)-[]->(a)”) (b) (a) (c) Then filter using vertex & edge data. paths.filter(“e1.delay > 20”)
- 29. 29
- 31. Graph algorithms Find importantvertices • PageRank 31 Find pathsbetweensets of vertices • Breadth-first search (BFS) • Shortest paths Find groupsof vertices(components, communities) • Connected components • Strongly connected components • Label Propagation Algorithm(LPA) Other • Triangle counting • SVDPlusPlus
- 32. 32
- 33. Algorithm implementations Mostly wrappers for GraphX • PageRank • Shortest paths • Connected components • Strongly connected components • Label Propagation Algorithm (LPA) • SVDPlusPlus 33 Some algorithms implemented usingDataFrames • Breadth-first search • Triangle counting
- 34. Saving & loading graphs Save & load the DataFrames. vertices = sqlContext.read.parquet(...) edges = sqlContext.read.parquet(...) g = GraphFrame(vertices, edges) g.vertices.write.parquet(...) g.edges.write.parquet(...) In the future... • SQL data sources for graph formats 34
- 35. APIs: Scala, Java, Python API available from all 3 languages à First time GraphX functionality hasbeen available to Java & Python users 2 missing items (WIP) • Java-friendliness is currently in alpha. • Python does not have aggregateMessages (for implementing your own graph algorithms). 35
- 36. Outline GraphFrames overview GraphFrames vs. GraphX and other libraries Details for power users Roadmap and resources 36
- 37. 2 types of graph libraries 37 Graph algorithms Graph queries Standard & custom algorithms Optimized for batch processing Motif finding Point queries &updates GraphFrames: Both algorithms &queries (but notpoint updates)
- 38. GraphFrames vs. GraphX 38 GraphFrames GraphX Builton DataFrames RDDs Languages Scala, Java, Python Scala Use cases Queries & algorithms Algorithms Vertex IDs Any type (in Catalyst) Long Vertex/edg e attributes Any number of DataFrame columns Any type (VD, ED) Return types GraphFrame or DataFrame Graph[VD, ED], or RDD[Long, VD]
- 39. GraphX compatibility Simple conversionsbetweenGraphFrames& GraphX. val g: GraphFrame = ... // Convert GraphFrame à GraphX val gx: Graph[Row, Row] = g.toGraphX // Convert GraphX à GraphFrame val g2: GraphFrame = GraphFrame.fromGraphX(gx) 39 Vertex & edgeattributes are Rows in order to handlenon-LongIDs Wrapping existing GraphX code: See Belief Propagation example: https://github.com/graphframes/graphframes/blob/master/src/main/scala/org/graphframes/examples/BeliefPropagation.scala
- 40. Outline GraphFrames overview GraphFrames vs. GraphXand other libraries Details for power users Roadmap and resources 40
- 41. Scalability Currentstatus • DataFrame-based parts benefitfrom DataFrame scalability + performance optimizations(Catalyst, Tungsten). • GraphX wrappers are as fast as GraphX (+ conversion overhead). WIP • GraphX hasoptimizationswhich are not yet ported to GraphFrames. • See nextslide… 41
- 42. WIP optimizations Join elimination • GraphFrame algorithms require lots of joins. • Not all joins are necessary Solution: • Vertex IDs serve as unique keys. • Tracking keys allows Catalyst to eliminate some joins. 42 For more info & benchmark results, see AnkurDave’s SSE 2016 talk. https://spark-summit.org/east-2016/events/graphframes-graph-queries-in-spark-sql/ Materializedviews • Data locality for common usecases • Message-passing algorithms often need “triplet view” (src,edge, dst) Solution: • Materialize specific views • Analogous to GraphX’s “replicated vertex view”
- 43. Implementing new algorithms 43 Method 2: Messagepassing aggregateMessages • Same primitive as GraphX • Specify messages & aggregation using DataFrame expressions Belief propagation example code Method 1: DataFrame & GraphFrame operations Motif finding • Series of DataFrame joins Triangle count • DataFrame ops + motif finding BFS • DataFrame joins & filters
- 44. Outline GraphFrames overview GraphFrames vs. GraphXand other libraries Details for power users Roadmap and resources 44
- 45. Current status Published • Open source (Apache 2.0) on Github https://github.com/graphframes/graphframes • Spark package http://spark- packages.org/package/graphframes/graphframes Compatible • Spark 1.4, 1.5, 1.6 • Databricks Community Edition Documented • http://graphframes.github.io/ 45
- 46. Roadmap • MergeWIP speed optimizations • Java API tests & examples • Migrate more algorithms to DataFrame-based implementations for greater scalability • Getcommunity feedback! 46 Contribute • Tracking issueson Github • Thanks to those who have already sent pull requests!
- 47. Resources for learning more User guide + API docs http://graphframes.github.io/ • Quick-start • Overview & examples for all algorithms • Alsoavailable as executablenotebooks: • Scala: http://go.databricks.com/hubfs/notebooks/3-GraphFrames-User-Guide-scala.html • Python: http://go.databricks.com/hubfs/notebooks/3-GraphFrames-User-Guide-python.html Blog posts • Intro: https://databricks.com/blog/2016/03/03/introducing-graphframes.html • Flight delay analysis: https://databricks.com/blog/2016/03/16/on-time-flight-performance- with-spark-graphframes.html 47
- 48. 48
- 49. Thank you! Thanks to • Denny Lee & Bill Chambers (demo) • Tim Hunter, Xiangrui Meng, Ankur Dave &others (GraphFrames development)