Data Structure Graph DMZ #DMZone
Download as PPTX, PDF2 likes533 views
The document presents an overview of data structure graphs, which are used to model relationships between entities in a data repository and visualize data flow across applications. It introduces different levels of data structure graphs, explores concepts from social network analysis, and discusses various methodologies for analyzing and optimizing data models, including Dijkstra's algorithm and centrality measures. Additionally, it emphasizes the importance of visual tools, like Gephi, for data analysis and encourages collaboration through sharing anonymized data for further exploration of these concepts.
Data Structure Graph DMZ #DMZone
- 1. Data Structure Graphs An overview Presentation by @dougneedham
- 2. Introduction @dougneedham Data Guy - Started as a DBA in the Marine Corps, evolved to Architect, now Data Scientist. Oracle, SQL Server, Cassandra, Hadoop, MySQL, Spark. I have a strong relational/traditional background. Perpetual Student Learning new things challenges our assumptions. Forces us to take a new perspective on “old” problems. Eventually maybe even shows us that there is a better way to solve a problem.
- 3. Introducing Data Structure Graphs Data Structure Graph Level 1 (DSG-L1)– This is roughly like an Entity Relationship Diagram (ERD) Tables are Vertices, Foreign Keys are Edges. A DSG-L1 can show you where you are going to have the most interesting query performance of your tables. Data Structure Graph Level 2 (DSG-L2) – Each Vertex in this graph is an application. Each Edge is data transfer. Roughly equivalent to what we used to call Data Flow diagrams. A DSG-L2 can show you where the most amount of work is going on in your Enterprise. Data Structure Graph Dependency (DSG-D) – Each vertex is a job, script, program, or process that is dependent on something happening in sequence before it can do its work. A DSG-D can show you the sequence of events that need to take place in order for something to be completed.
- 4. Definition A Data Structure Graph is a group of atomic entities that are related to each other, stored in a repository, then moved from one persistence layer to another, rendered as a Graph. A group of atomic entities. Related to each other. Stored in a repository. Moved from one persistence layer to another. Rendered as a Graph.
- 5. In summary: Social Network analysis applied to data modeling. Data modeling is a topic we are all familiar with here at data modeling zone. Social Network analysis is, perhaps, something new. So a little background on the topic we may not be familiar with.
- 6. What is Social Network Analysis? “Social network analysis (SNA) is a strategy for investigating social structures through the use of network and graph theories. It characterizes networked structures in terms of nodes (individual actors, people, or things within the network) and the ties or edges (relationships or interactions) that connect them. Examples of social structures commonly visualized through social network analysis include social media networks, friendship and acquaintance networks, kinship, disease transmission, and sexual relationships. These networks are often visualized through sociograms in which nodes are represented as points and ties are represented as lines.” – Wikipedia https://en.wikipedia.org/wiki/Social_network_analysis
- 7. Example From wiki: "Kencf0618FacebookNetwork" by Kencf0618 - Own work. Licensed under CC BY-SA 3.0 via Wikimedia Commons - https://commons.wikimedia.org/wiki/File:Kencf0 618FacebookNetwork.jpg#/media/File:Kencf061 8FacebookNetwork.jpg
- 8. A little History The 7 Bridges of Konigsberg Every tome on Graph theory or Network analysis devotes a small portion of there time to the 7 Bridges of Konigsberg. If I don’t cover this with you, the gods of mathematics will strike me down, and never allow me to do analysis again in the future.
- 9. The Bridges
- 10. The Problem Folks enjoyed there Sunday afternoon strolls across the bridges, but occasionally people would wonder if one particular route was more efficient than another. Eventually Leonhard Euler was brought into the debate about the efficiency problem. Euler used Vertices to represent the land masses and edges (or arcs, at the time) to represent bridges. He realized the odd number of edges per vertex made the problem unsolvable. Sarada Herke provides for one of the best explanations of the solution Solution to Konigsberg Basically the solution is that a vertex must have an even number of edges in order to make it possible to start from one vertex, and arrive at the point of origin without crossing any edge twice. Essentially, the number of bridges must be an even number. (more details in the above video) And here is the cool thing about mathematicians. If we tell you something is impossible, we have to tell you why in a way you can understand it. But he also invented the branch of mathematics today we call Graph Theory. http://en.wikipedia.org/wiki/Leonhard_Euler
- 11. A few terms Stand back, we are going to talk about math! Basically we are talking about a bunch of dots joined together by lines Vertex – Dot on a graph Edge – Line connecting the two points Edge_Label – this is a term I coined originally related to Data Structure Graphs that helps trace a path. If you label your edges, and you have multiple edges with the same label in a Graph you can quite easily identify walks, paths, and cycles through your graph. A lot of things are networks if you look at them the right way. Mark Newman has done a number of really cool presentations, available on YouTube about Network analysis. https://www.youtube.com/watch?v=lETt7IcDWLI
- 12. More terms What is a path? Shortest path – How are two vertices connected? Longest Path – Tracing the flow of an interesting item through a large collection of applications. Directed Graphs – or Digraphs If you rearrange things how does the layout affect understanding? This is not just data visualization, it can also be used for prediction. https://www.youtube.com/watch?v=rwA-y-XwjuU
- 13. Final terms Centrality – Hub and Authority This is almost a whole topic by itself, since there are different types of Centrality: Degree Centrality, Eigenvector Centrality, PageRank, etc… Longest Path – Tracing the flow of an interesting item through a large collection of applications. Power law. What is a path? Centrality – Hub and Authority This is almost a whole topic by itself, since there are different types of Centrality: Degree Centrality, Eigenvector Centrality, PageRank, etc… Transitivity Homophily – how things are similar Directed Graphs – or Digraphs Contagion – How do things “spread” through a network? Let’s rearrange things, how does the layout affect understanding? Order of a graph – number of vertices Size of the graph – number of edges This is not just data visualization, it can also be used for prediction. https://www.youtube.com/watch?v=rwA-y-XwjuU
- 14. The Math doesn’t change. One thing I like about Graphs – The Math does not change. The math behind Graph theory can be a little intense, but it does not change regardless of the scale of the graph. Once you understand how to “do the math” on a small graph, those same Math's apply to a Graph whether it is a graph of the people in this room, or a graph of the people on this planet.
- 15. Before we get to the analysis we must collect data. Dbeaver can reverse engineer an ERD. Point it at the source system, select a few options, then you have a diagram. I wrote a small piece of Python code to translate the XML to a file suitable for import into Gephi. One small caveat: the Foreign keys have to be defined for Dbeaver to work. If the foreign keys are not defined the output file will need to be modified. Also, some aggregate or summary tables may not help your visualization. This is subjective, so it is at the discretion of the person reviewing the diagram. If you remove tables from the graph, please provide documentation such that the visualization can be compared to the reality of your data model with no discrepancies. Url for Dbeaver is here: https://dbeaver.jkiss.org/ (This section is a little hand-wavy I know but the tool, or method for creating the file for import into Gephi is largely irrelevant.)
- 16. Gephi http://gephi.github.io/ From the website: “Gephi is an interactive visualization and exploration platform for all kinds of networks and complex systems, dynamic and hierarchical graphs.” We are going to use data from generated from my book: Data Structure Graphs. These are inspired by my experience consulting, but do not represent an actual data model, or etl process. The following slides are for a DSG Level 2 (Etl process).
- 17. Gephi Startup
- 18. New Project, Data Table, Import data.
- 19. Load as “Edges Table” Source, Target (required)
- 20. Choose Create Missing Nodes
- 21. After a few calculations and layout runs
- 22. PageRank – Which application is most important?
- 23. A few more tweaks
- 24. Where is that Node with the highest PageRank?
- 25. Now things get interesting: New metrics for our data model follow. Remember all those metrics we defined earlier? Here are many of them:
- 26. Data Table
- 27. Configure Labels
- 28. Labeled by degree count
- 29. Change some of the coloring
- 30. Visualization
- 31. Export to Excel
- 32. Finally, here we are. Within a Data Architecture there are lots of moving pieces. ETL, FTP, SFTP, Web- Services, External data feeds. Data moving into Data Marts, and Data Warehouses. Data Moving between applications. Let’s imagine how to visualize this using the information we just gained.
- 33. Data Structure Graphs Today, there are a few tools like ERWin, and SQL Developer that begin to organize visualizations in this manner. Very few of them allow you to perform analysis on the visualization. As you find new tools that do this, please let me know. I would love to evaluate those tools and see what interesting metrics can be arrived at from new tools.
- 34. Dijkstra's algorithm Some of you may have heard of Dijkstra’s algorithm. It is a method for finding the shortest path between two nodes on a Graph. This is a great optimization technique, but what if you need to find the longest path? What “Edge_Label” has the most influence on my organization? Iterate through each Edge_Label, create a subgraph that consists of only the nodes this Edge_Label touches, then calculate the diameter of that Graph. The Edge_Label that is longest has the most “impact” on your organization. This is mostly applied to Data Structure Graph Level 2.
- 35. Now let’s answer some questions. Which table is “most important” to ensure you are importing to build a data warehouse? The tables with the higher centrality measures. For an operational system these will also be the tables that have the most queries written against them. These will be your bottlenecks for any system. Is this data model optimized for reading or writing? What is the density of the data model? The higher density is optimized for write, lower density is optimized for read.
- 36. Barabasi-Albert model and Scale free networks. Preferential attachment. There are a few different models available for analysis and prediction of networks. A Barabsi-Albert model can be summarized as a “rich get richer” model. In other words, the more connected a node is, when new nodes are added, they are more than likely connected to these well connected nodes. This suspiciously sounds similar to our data modeling concepts related to conformed dimensions. My suspicion is there are many data models that fit this model. Please send me some anonymized data models. I want to research this more.
- 37. Some theoretical thoughts. Let’s assume we have an equation for the growth of every table we have collected from our little topological study above(more on this in a couple slides). Let us further assume we have a graph of the same tables. Can you do anything interesting with this? The derivative of each equation shows us the growth rate of the table. What happens if we plug that derivative in the entropy equation for the graph? What would this represent? Could this be considered an valuation method? A way to put a dollar value on a data model? If you try it, let me know what you find out.
- 38. Apply the theory. Using a few metrics from each table we can do some clustering. Take the number of columns of a table, the centrality measure, and the growth rate you have a vector for each table. Doing some simple cosine similarity on these vectors will tell you mathematically which tables are similar. Is this finding consistent with expectations? If not should the model be adjusted? What does this result say to you?
- 39. Deriving the growth rate of each table. Little R demonstration to follow. Using a design methodology like the data vault mandates that every table have date timestamps for when the data is loaded. Collect how many records are loaded per day. A calculation that represents the growth formula for each table can be derived with R. Using the growth rate, centrality, and the width of a table (number of columns) you can do cosine similarity to determine the tables that are mathematically similar to each other. Using this information you may be able to reallocate the infrastructure that the data warehouse sits on. Is every table stored on the same disk storage media? Does it need to be? How about caching? Using these metrics alone you can make a well informed decision about your storage platform. The following image is a small topological representation of this process. This is still slightly theoretical, and I welcome having a conversation with anyone that may want to know more. Again, send me anonymized data. Hopefully along with the Data Structure Graph you generated from your data.
- 40. This is what the topology may look like.
- 41. Consider the following: If you need assistance, contact me directly (I am easy to find @dougneedham) Network/Graph Analysis is cool. It can show you some interesting things about your data that you may not have considered.
- 42. What did I leave out? Graphs that change over time – What happens when you remove a single Edge or Vertex? Comparing two networks – If you have the same number of edges and nodes, are two graphs the same? Contagion – How will data spread through the network. (Since a DSG represents different types of Edges based on Edge_Label, Contagion should not affect the entire network). This is also commonly known as data lineage. If you don’t have a tool that does it, with a bit of metadata management this can be derived from a Data Structure Graph Level 2
- 43. Other Analysis What else can be done with Social Network Analysis? How about risk exposure to banks? http://www.federalreserve.gov/newsevents/speech/yellen20130104a.htm
- 44. A little history
- 45. One other cool bit of Math How many reports can your dimensional data model support? Do you have the situation where people want to create a project out of a report, rather than do a proper data model design up front? Here is some help. The upper bound of the total number of reports that a conformed dimension data model can support is calculated by: Calculate the number of selectable columns in each dimension (2 𝑐 − 1) Create the adjacency matrix for the dimensions to facts A bit of multiplication. More details here: http://bit.ly/MeasuringDimensionalModels
- 46. Graphs are Cool! Help me. Please send me anonymized data. In order to present more about how the mathematics of Graph theory, and social network analysis can be applied in general to the application of data modeling, I need more data. This is a fascinating topic, if you want to reach out to me directly I can be reached at: [email protected] Here is my GitHub for the code and data from the book, and examples: http://bit.ly/DataStructureGraph_github
- 47. https://dougneedham.shinyapps.io/DataStructureGraph Hard to see, I know, but the top diagram is the “master graph”, the bottom image is a single Edge_Label. You can see how an individual data entity flows through an organization.
- 48. My book Goes through a number of examples for doing an Graph analysis of a fictional organization.
- 49. Final Thoughts – Questions?