Download as PDF, PPTX
Uploaded byMathieu Bastian
Visualize Big Graph Data
The document discusses big graph data and data visualization. Some key points: - Graphs are useful for solving problems across many domains like disease spread, recommendations, and power grids. - Graphs can be small, medium, large, explicit, or implicit based on how the network is constructed from data. - Visualization is a natural tool for exploring and understanding complex graph data through interaction and iteration. - Open source tools like Gephi and Sigma.js can be used to visualize and analyze graph data.
Visualize Big Graph Data
- 1. M AT HI E U B A S T I A N D ATA V I S U A L I Z AT I O N S U M M I T, 1 SAN FRANCISCO, APRIL 11-12, 2013
- 2. BIG GRAPH DATA • The story of big graph data is just starting • BIG GRAPH DATA DATA VISUALIZATION SUMMIT 2 2
- 3. BIG GRAPH DATA • The story of big graph data is just starting • BIG GRAPH DATA BIG DATA GRAPHS DATA VISUALIZATION SUMMIT 3 3
- 4. BIG GRAPH DATA • The story of big graph data is just starting • BIG GRAPH DATA BIG DATA GRAPHS DISTRIBUTED SYSTEMS COMPLEX STORAGE DATABASES INDEXATION LARGE DATASETS ALGORITHM CLOUD COMPUTING HADOOP ANALYTICS REAL-TIME VISUALIZATION DATA VISUALIZATION SUMMIT 4 4
- 5. BIG GRAPH DATA • The story of big graph data is just starting • BIG GRAPH DATA BIG DATA GRAPHS DISTRIBUTED SYSTEMS COMPLEX STORAGE DATABASES INDEXATION LARGE DATASETS ALGORITHM CLOUD COMPUTING HADOOP ANALYTICS REAL-TIME VISUALIZATION DATA VISUALIZATION SUMMIT 5 5
- 6. BIG DATA • “The Petabyte age” • All industries and domains can leverage big data Health Government Finance Technology • Big Data => Big Problems • Focusing on building the technology to handle big data, and big graph data (ex: graph databases) • Seeking efficient analysis of ever more complex systems DATA VISUALIZATION SUMMIT 6 6
- 7. GRAPHS • Graphs are everywhere, and it’s easy to collect graph data • The world is more complex and interconnected that we thought Source: Collective Dynamics of Small-World Networks, D Watts, S Strogatz, Nature 393, 440-442 DATA VISUALIZATION SUMMIT 7 7
- 8. NETWORK SCIENCE • The study of graphs has been exploding in the last 15 years • Networks have properties and patterns one can study • Robustness – How a network is resistant to random attacks? • Contagion – How fast a disease or gossip spread in a network? • Communities – How many communities exist in a network? • Centrality – Who is the most central individual in a network? • If you read one of these books, you understand Network Science DATA VISUALIZATION SUMMIT 8 8
- 9. GRAPHS HELP SOLVEPROBLEMS • Saddam Hussein Network (2003) The Universe C. Wilson. Searching for Saddam: a five-part series on how the US military used social networking to capture the Iraqi dictator. 2010. www.slate.com/ id/2245228/. DATA VISUALIZATION SUMMIT 9 9
- 10. GRAPHS HELP SOLVEPROBLEMS • Predicting and controlling infectious disease Naoki Masuda, Petter Holme - Predicting and controlling infectious disease The Universe epidemics using temporal networks. http://f1000.com/prime/reports/b/5/6/ Haraldsdottir S, Gupta S, Anderson RM: Preliminary studies of sexual networks in a male homosexual community in Iceland. J Acquir Immune Defic Syndr. 1992, 5:374–81. DATA VISUALIZATION SUMMIT 10 1 0
- 11. GRAPHS HELP SOLVEPROBLEMS • Recommendation systems The Universe Credit: http://markorodriguez.com/2011/09/22/a-graph-based-movie-recommender-engine/ DATA VISUALIZATION SUMMIT 11 1 1
- 12. GRAPHS HELP SOLVEPROBLEMS • Recipe recommendation using ingredient networks The Universe Credit: http://www.ladamic.com/wordpress/?p=294 1 DATA VISUALIZATION SUMMIT 21 2
- 13. GRAPHS HELP SOLVEPROBLEMS • Power grid The Universe Credit: http://www.npr.org/templates/story/story.php?storyId=110997398 DATA VISUALIZATION SUMMIT 13 1 3
- 14. SMALL GRAPHS • Famous “Zachary’s Karate Club” study in 1977 only involved 34 nodes. • It could be drawn by hand on paper The Universe Zachary’s Karate Club (1977) W. W. Zachary, An information flow model for conflict and fission in small groups, Journal of Anthropological Research 33, 452-473 (1977). DATA VISUALIZATION SUMMIT 14 1 4
- 15. MEDIUM GRAPHS • Your own Facebook or LinkedIn social network • The Harlem Shake: Anatomy of a Viral Meme The Universe Gilad Lotan. http://www.huffingtonpost.com/gilad-lotan/the-harlem-shake_b_2804799.html DATA VISUALIZATION SUMMIT 15 1 5
- 16. LARGE GRAPHS • The Internet Map (~350 000 domains) • DBPedia (~290M relationships) • Friendster Social Network dataset* (1.8B edges) The Universe Internet Map (http://internet-map.net) * http://snap.stanford.edu/data/index.html DATA VISUALIZATION SUMMIT 16 1 6
- 17. IMPLICIT GRAPHS • Graphs can be explicit or implicit • Explicit: The network exists in nature (Social Network, Food Webs, Airlines Network) • Implicit: The network is derived from other data (Word networks, co- authorship) • Example of an implicit graph: • A set of documents have a set of tags • One can create a link when two tags are on the same document • Aggregate all links across all documents DATA VISUALIZATION SUMMIT 17 1 7
- 18. SIMILARITY GRAPHS • Graphs of all the co-occurrences between LinkedIn Skills (2011) DATA VISUALIZATION SUMMIT 18 1 8
- 19. VISUALIZATION • Visualization and statistics are the two basic toolkits one can use on graphs • Complex questions are asked when studying graphs • Easy • Min, max, average, quartiles Excel can do this! • Exact queries, search • Harder • Patterns, trends, correlations • Changes over time, context • Anomalies, data errors Visualization can do this! • Geographical representation DATA VISUALIZATION SUMMIT 19 1 9
- 20. GRAPH VISUALIZATION • Due to the size of graphs and the complexity of questions, visualization is the natural tool to understand what’s going on “ We are more easily persuaded by the reasons we ourselves discover than by those which are given to us by others.” Blaise Pascal Let me play with the data! Direct manipulation DATA VISUALIZATION SUMMIT 20 2 0
- 21. DATA EXPLORATION ANDINTERACTION • Use visualization and statistics to discover new hypothesis • Exploratory data analysis “The greatest value of a picture is when it forces us to notice what we never expected to see.” John Tukey • The user interface is centered around the human • Empowers the user to understand the structure and patterns in the data • The machine augments the human • How? • Overview and details, zoom and pan interface • Interactive, direct-manipulation DATA VISUALIZATION SUMMIT 21 2 1
- 22. MAP YOUR DATA • Iterative process to transform relational data into a map • Use color, size and position to highlight, group and set up a hierarchy DATA VISUALIZATION SUMMIT 22 2 2
- 23. FROM INFORMATION TOKNOWLEDGE • Exploring networks interactively & iterating often provide “Eureka” moments for domain experts Eureka DATA VISUALIZATION SUMMIT 23 2 3
- 24. BIG GRAPH DATA • Big graph data doesn’t necessarily mean you’re visualizing or analyzing a large graph • Small graphs can be extracted from large graphs and analyzed • Small graphs can be extracted from non-graph data as well • Graphs are just nodes and relationships after all • Example: Adverse Drug Event Analysis with Hadoop, R, and Gephi (Josh Wills, Cloudera, 2012) DATA VISUALIZATION SUMMIT 24 2 4
- 25. GEPHI • Built to solve large graph visualization problems. • Open source tool for Windows, Mac OS X and Linux • Large international community involved • The latest version has been downloaded > 100,000 times • Extensible with plug-ins • Available at http://gephi.org DATA VISUALIZATION SUMMIT 25 2 5
- 26. GEPHI DATA EDITION VISUAL MAPPING FILTER VISUALIZATION STATISTICS LAYOUT TIMELINE DATA VISUALIZATION SUMMIT 26 2 6
- 27. SIGMA.JS • Open-source lightweight JavaScript library to draw graphs • Uses HTML5 Canvas • Display dynamically graphs that can be generated on the fly • Available at http://sigmajs.org Sigma.js v0.1 DATA VISUALIZATION SUMMIT 27 2 7
- 28. SUMMARY • Big graph data = Relational Big Data • Graphs are everywhere! • Graphs have fascinating structure and patterns one can analyze • Visualization is a natural tool for such complex data and complex questions • On graphs, visualization done right allows interaction and iteration. Play. • The hard part is to extract a small or medium graph from big data • Open source tools like Gephi or Sigma.js are a good start DATA VISUALIZATION SUMMIT 28 2 8
- 29. Become a graphevangelist! QUESTIONS? Mathieu Bastian (@mathieubastian) DATA VISUALIZATION SUMMIT 29 2 9
- 30. REFERENCES & LINKS Join the Social Network Analysis class by Lada Adamic on Coursera Sigma.js, Alexis Jacomy and al. https://www.coursera.org/course/sna http://sigmajs.org Support the Gephi Consortium Linked: How Everything Is Connected to Everything Else and What It http://consortium.gephi.org Means, Albert-Laszlo Barabasi http://www.amazon.com/gp/product/0452284392/ Computational Information Design, Ben Fry (2004) http://benfry.com/phd/ Six Degrees: The Science of a Connected Age, Duncan J. Watts http://www.amazon.com/gp/product/0393325423/ The Atlas of Economic Complexity, Harvard's Center for International Development (CID) and the MIT Media Lab Nexus: Small Worlds and the Groundbreaking Science of Networks, http://atlas.media.mit.edu/ Mark Buchanan http://www.amazon.com/gp/product/0393324427 The Mesh of Civilizations and International Email Flows, Bogdan State, Patrick Park, Ingmar Weber, Yelena Mejova, Michael Macy Connected: The Surprising Power of Our Social Networks and How They http://arxiv.org/abs/1303.0045 Shape Our Lives, Nicholas A. Christakis and James H. Fowler http://www.amazon.com/dp/product/0316036137 The Human Disease Network, Goh K-I, Cusick ME, Valle D, Childs B, Atelier Iceberg – Gephi Vidal M, Barabási A-L (2007) http://www.slideshare.net/ateliericeberg/gephi-17680699 http://www.pnas.org/content/104/21/8685.full Adding Value through graph analysis using Titan and Faunus, Matthias What does your intranet look like? Broecheler http://intranetdiary.blogspot.co.uk/2012/11/network-visualisation.html http://www.slideshare.net/knowfrominfo/titan-talk-ebaymarch2013 Recipe recommendation using ingredient networks, Chun-Yuen Teng, Yu- Network Maps Board on Pinterest, Mathieu Bastian Ru Lin, Lada A. Adamic http://pinterest.com/mathieubastian/network-maps/ http://arxiv.org/abs/1111.3919 Network Science Book, Albert-László Barabási US Presidents Inaugural Speeches 1969-2013 Text Network Analysis http://barabasilab.neu.edu/networksciencebook http://noduslabs.com/cases/presidents-inaugural-speeches-text- network-analysis/ Adverse Drug Event Analysis with Hadoop, R, and Gephi, Cloudera https://github.com/cloudera/ades 10 Reasons Why We Visualise Data http://www.slideshare.net/Facegroup/10-reasons-why-we-visualise-data DATA VISUALIZATION SUMMIT 30 3 0