Gephi, Graphx, and Giraph
Download as PPTX, PDF•2 likes•2,167 views
The document discusses graph theory and its relevance in data analysis, particularly through the lens of network analysis. It highlights the contributions of mathematician Leonhard Euler and introduces concepts such as vertices, edges, and various types of graphs while showcasing tools like Gephi for visualization. Additionally, it presents two types of data structure graphs that can be useful for understanding query performance and data flow within enterprises.
Gephi, Graphx, and Giraph
- 1. Graph Theory at work [email protected]
- 2. • @dougneedham • Data Guy - Started as a DBA in the Marine Corps, evolved to Architect, now aspiring Data Scientist. • Oracle, SQL Server, Cassandra, Hadoop, MySQL. • I have a strong relational/traditional background. • Perpetual Student • Learning new things challenges our assumptions. Forces us to take a new perspective on “old” problems. Eventually maybe even shows us that there is a better way to solve a problem.
- 3. • Stand back, we are going to talk about math! • Basically we are talking about a bunch of dots joined together by lines • Vertex – Dot on a graph • Edge – Line connecting the two points • Triangle – 3 Vertices, 3 Edges • Square – 4 Vertices, 4 edges • Open Triangle - 3 Vertices, 2 edges • A lot of things are networks if you look at them the right way. • Mark Newman has done a number of really cool presentations, available on Youtube about Network analysis. • https://www.youtube.com/watch?v=lETt7IcDWLI
- 5. • The 7 Bridges of Konisberg • Every tome on Graph theory or Network analysis devotes a small portion of there time to the 7 Bridges of Konisberg. • If I don’t cover this with you, the gods of mathematics will strike me down, and never allow me to do analysis again in the future.
- 7. • Folks enjoyed there Sunday afternoon strolls across the bridges, but occasionally people would wonder if one particular route was more efficient than another. • Eventually Leonhard Euler was brought into the debate about the efficiency problem. • Euler used Vertices to represent the land masses and edges (or arcs, at the time) to represent bridges. He realized the odd number of edges per vertex made the problem unsolvable. • And here is the cool thing about mathematicians. If we tell you something is impossible, we have to tell you why in a way you can understand it. But he also invented the branch of mathematics today we call Graph Theory. • http://en.wikipedia.org/wiki/Leonhard_Euler
- 8. • http://gephi.github.io/ • From the website: “Gephi is an interactive visualization and exploration platform for all kinds of networks and complex systems, dynamic and hierarchical graphs.” • To get this yourself go into Facebook and search for: Netvizz. (You have to authorized it. You can un-authorized it later) • Click the application. • Click “personal network” • Click Start • Download your gdf file • Quick Demo:
- 9. • Shortest path – How are two vertices connected? • What is a path? • Centrality • Transitivity • Homophily • Directed Graphs – or Digraphs • Contagion – How do things “spread” through a network? • Let’s rearrange things, how does the layout affect understanding? • This is not just data visualization, it can also be used for prediction. https://www.youtube.com/watch?v=rwA-y-XwjuU
- 10. • Requires Spark, which is not a bad deal. • Jump to Demo • http://ampcamp.berkeley.edu/big-data-mini-course/graph-analytics-with-graphx. html
- 11. • Giraph, I haven’t really done as much with as I wanted to do. Perhaps a later presentation with a more detailed example comparing GraphX with Giraph.
- 12. • I started doing some analysis some time ago using Graph models to understand metadata. • I came up with two types of Graphs: • Data Structure Graph Level 1 – This is roughly like an Entity Relationship Diagram (ERD) Tables are Vertices, Foreign Keys are Edges. • Data Structure Graph Level 2 – Each Vertex in this graph is an application. Each Edge is data transfer. Roughly equivalent to what we used to call Data Flow diagrams.
- 13. • A DSG Level 1 can show you where you are going to have the most interesting query performance of your tables. • A DSG Level 2 can show you where the most amount of work is going on in your Enterprise.
- 14. • Network/Graph Analysis is cool. • It can show you some interesting things about your data. • Some things to consider. • Some thought needs to be put into how the raw data is organized for a Graph Analysis. • Directed graph, undirected, bigraph? Some up front setup work needs to be done. • Tools help with the detailed calculations, and show the paths, walks, etc. • However, due thought should be put towards a network analysis project.