![(s:Entity)-[r:CREATED]->(t:Entity)](https://image.slidesharecdn.com/2017-01-08-scalingtribalknowledge-170209020857/85/2017-01-08-scaling-tribalknowledge-72-320.jpg)
2017-01-08-scaling tribalknowledge
- 1. Scaling Tribal Knowledge CHRIS WILLIAMS / JOHN BODLEY / FEB 8, 2017 / BIG DATA APPLICATION MEETUP
- 2. The problem
- 3. tribal knowledge |ˈtrībəl ˈnäləj | noun Tribal knowledge is any unwritten information that is not commonly known by others within a company
- 4. As Airbnb grows so do the challenges around the volume, complexity, and obscurity of data
- 5. In a large and complex organization, with a sea of data resources, users struggle to find the right data
- 6. Data is often siloed and lacks context
- 7. I’m a recovering Data Scientist who wants to democratize data; automate common workflows, surface relevant information, and provide context
- 9. Tables in our Hive data warehouse 100k
- 10. Data resources Beyond the data warehouse
- 11. > 6,000 Superset charts and dashboards Data resources Beyond the data warehouse
- 12. > 6,000 Superset charts and dashboards > 5,000 Experiments and metrics Data resources Beyond the data warehouse
- 13. > 6,000 Superset charts and dashboards > 5,000 Experiments and metrics > 4,000 Tableau dashboards and workbooks Data resources Beyond the data warehouse
- 14. > 6,000 Superset charts and dashboards > 5,000 Experiments and metrics > 4,000 Tableau dashboards and workbooks > 1,000 Knowledge posts Data resources Beyond the data warehouse
- 15. With many more data sources and data types to love
- 16. With many more data sources and data types to love
- 19. > 3,000 Airbnb employees
- 20. Portland San Francisco Los Angeles Toronto New York Miami Sao Paulo Dublin London Paris Barcelona Berlin Milan Copenhagen New Delhi Seoul Beijing Tokyo Sydney Singapore Washington, DC > 20 Offices around the world
- 21. The mandate
- 22. To democratize data and empower Airbnb employees to be data- informed by aiding with data exploration, discovery, and trust
- 23. The concept
- 24. Search…
- 25. It should be fairly evident what we feed into the search indices
- 28. But are we missing something?
- 30. The relevancy of relationships Nodes and relationships have equal standing created consumedSpoke 3
- 38. The construction
- 41. Databases 5 APIs 3 Airflow DAG 1 We leverage all these data resources to build a graph comprising of nodes and relationships The Airflow DAG is run everyday and the output is stored in Hive
- 43. We gather over 10,000 thumbnails from the Tableau API, Knowledge Repo database, and Superset screenshots
- 44. The winding data path Airflow Data transfer Python Graph datastore py2neo Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 45. The winding data path Airflow Data transfer Python Graph datastore py2neo Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 46. The winding data path Airflow Data transfer Python Graph datastore py2neo Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 47. The winding data path Airflow Data transfer Python Graph datastore py2neo Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 48. The winding data path Airflow Data transfer Python Graph datastore py2neo Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 49. The winding data path Airflow Data transfer Python Graph datastore py2neo Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 50. The winding data path Airflow Data transfer Python Graph datastore py2neo Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 51. The winding data path Airflow Data transfer Python Graph datastore py2neo Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 52. The winding data path Airflow Data transfer Python Graph datastore py2neo Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 53. The winding data path Airflow Data transfer Python Graph datastore py2neo Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 54. The winding data path Airflow Data transfer Python Graph datastore py2neo Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 55. The winding data path Airflow Data transfer Python Graph datastore py2neo Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 56. The winding data path Airflow Data transfer Python Graph datastore py2neo Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 57. The winding data path Airflow Data transfer Python Graph datastore py2neo Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 58. The winding data path Airflow Data transfer Python Graph datastore py2neo Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 59. Why we choose Neo4j for our database The main reasons
- 60. Logical Given our data is represented as a graph it is logical to use a graph database to store the data Why we choose Neo4j for our database The main reasons
- 61. Logical Given our data is represented as a graph it is logical to use a graph database to store the data Nimble Performance wins when dealing with connected data versus relational databases Why we choose Neo4j for our database The main reasons
- 62. Logical Given our data is represented as a graph it is logical to use a graph database to store the data Nimble Performance wins when dealing with connected data versus relational databases Popular It is the world’s leading graph database and the community edition is free Why we choose Neo4j for our database The main reasons
- 63. Logical Given our data is represented as a graph it is logical to use a graph database to store the data Nimble Performance wins when dealing with connected data versus relational databases Popular It is the world’s leading graph database and the community edition is free Integrative It integrates well with Python and Elasticsearch Why we choose Neo4j for our database The main reasons
- 64. The Neo4j and Elasticsearch symbiotic relationship Courtesy of two GraphAware plugins
- 65. The Neo4j and Elasticsearch symbiotic relationship Courtesy of two GraphAware plugins Neo4j plugin Provides bi-directional integration which transparently and asynchronously replicate data from Neo4j to Elasticsearch
- 66. The Neo4j and Elasticsearch symbiotic relationship Courtesy of two GraphAware plugins Neo4j plugin Provides bi-directional integration which transparently and asynchronously replicate data from Neo4j to Elasticsearch Elasticsearch plugin Enables Elasticsearch to consult with the Neo4j database during a search query to enrich the search rankings by leveraging the graph topology
- 70. The schema
- 71. createds t
- 73. :Entity :Org :Group :User :Superset :Slice:Dashboard Node label hierarchy :Hive :Schema :Table
- 77. Efficient data retrieval and uniqueness Restrictions and workarounds with the Neo4j schema
- 78. Efficient data retrieval and uniqueness Restrictions and workarounds with the Neo4j schema Indexes Neo4j provides indexes for efficient data retrieval similar to a RDMS, however they are only defined for a single label
- 79. Efficient data retrieval and uniqueness Restrictions and workarounds with the Neo4j schema Indexes Neo4j provides indexes for efficient data retrieval similar to a RDMS, however they are only defined for a single label Uniqueness Constraints Ensures that properties are unique for all nodes for a specific single label
- 80. Efficient data retrieval and uniqueness Restrictions and workarounds with the Neo4j schema Indexes Neo4j provides indexes for efficient data retrieval similar to a RDMS, however they are only defined for a single label Uniqueness Constraints Ensures that properties are unique for all nodes for a specific single label GraphAware UUID plugin Transparently assigns a globally unique UUID property to newly created elements which cannot be changed or deleted
- 82. The web app
- 83. The web app
- 84. Designing the user experience and interface of a data tool should not be an afterthought
- 85. Designing the user experience and interface of a data tool should not be an afterthought
- 86. Technical data power user; the epitome of a tribal knowledge holder Daphne Data User personas Less data literate; needs to keep tabs on her team’s resources Manager Mel New employee or new team; has no idea what’s going on Nathan New
- 87. Designing for data exploration, discovery, and trust Company dataSearch Resource details &meta-data User data Group data
- 88. Company dataSearch User data Group data Resource details &meta-data
- 89. Company dataSearch User data Group data Resource details &meta-data
- 90. Search Resource details &meta-data Company dataUser data Group data
- 91. Search Resource details &meta-data Company dataUser data Group data Google-esque search filters
- 92. Search Resource details &meta-data Company dataUser data Group data Google-esque search filters Resource details & meta-data
- 93. Search Resource details &meta-data Company dataUser data Group data Google-esque search filters Resource details & meta-data Context, context, & context
- 94. Search Resource details &meta-data Company dataUser data Group data
- 95. Search Resource details &meta-data Company dataUser data Group data Description, external link, social
- 96. Search Resource details &meta-data Company dataUser data Group data Meta-data & consumption Description, external link, social
- 97. Search Resource details &meta-data Company dataUser data Group data Surface relationships, everything’s a link to promote exploration Meta-data & consumption Description, external link, social
- 98. Column details & value distributions Table lineage Enrich meta-data on the fly Search Resource details &meta-data Company dataUser data Group data
- 99. Column details & value distributions Table lineage Enrich meta-data on the fly Search Resource details &meta-data Company dataUser data Group data
- 100. Search Resource details &meta-data Company dataUser data Group data
- 101. Search Resource details &meta-data Company dataUser data Group data
- 102. Search Resource details &meta-data Company dataUser data Group data
- 103. Search Resource details &meta-data Company dataUser data Group data
- 104. User details & meta-data Search Resource details &meta-data Company dataUser data Group data
- 105. User details & meta-data What they make, what they consume Search Resource details &meta-data Company dataUser data Group data
- 106. Former employees also hold tribal knowledge Search Resource details &meta-data Company dataUser data Group data
- 107. Search Resource details &meta-data Company dataUser data Group data
- 108. Group overview Search Resource details &meta-data Company dataUser data Group data
- 109. Group overview Search Resource details &meta-data Company dataUser data Group data Pinterest-like curation
- 110. Group overview Search Resource details &meta-data Company dataUser data Group data Basic organization functionality Pinterest-like curation
- 111. Search Resource details &meta-data Company dataUser data Group data Curated + Popular content
- 112. Search Resource details &meta-data Company dataUser data Group data Curated + Popular content Thumbnails for maximum context
- 113. Search Resource details &meta-data Company dataUser data Group data Pinning flow from resource page Edit mode / draggable grid
- 114. Search Resource details &meta-data Company dataUser data Group data Pinning flow from resource page Edit mode / draggable grid
- 115. ???? ?? Employees can feel disconnected from Company-level metrics Search Resource details &meta-data Company dataUser data Group data
- 116. The technology stack Application + dependencies DOM Testing eslint enzyme mocha chai Application state Styling khan/aphrodite
- 117. The challenges
- 118. The challenges
- 119. The challenges Complex dependencies An umbrella data tool is vulnerable to changes in upstream resource dependencies
- 120. The challenges Complex dependencies An umbrella data tool is vulnerable to changes in upstream resource dependencies Data-dense design Balancing simplicity and functionality is hard; most internal design resources are not made for data-rich apps
- 121. The challenges Complex dependencies An umbrella data tool is vulnerable to changes in upstream resource dependencies Data-dense design Balancing simplicity and functionality is hard; most internal design resources are not made for data-rich apps Graph merging Non-trivial Git-like merging of (daily or real- time) graph updates
- 122. The challenges Complex dependencies An umbrella data tool is vulnerable to changes in upstream resource dependencies Data-dense design Balancing simplicity and functionality is hard; most internal design resources are not made for data-rich apps Graph flickering Transient relationships should not create “flickering” artifacts Graph merging Non-trivial Git-like merging of (daily or real- time) graph updates
- 123. The future
- 124. The future
- 125. The future New resource types A/B tests, logging schemas, SQL queries, etc.
- 126. The future New resource types A/B tests, logging schemas, SQL queries, etc. Certified content Use certification to build trust and enable users to filter through a sea of stale content
- 127. The future New resource types A/B tests, logging schemas, SQL queries, etc. Certified content Use certification to build trust and enable users to filter through a sea of stale content Alerts& recommendations Move from active exploration to deliver relevant updates and content suggestions
- 128. The future New resource types A/B tests, logging schemas, SQL queries, etc. Certified content Use certification to build trust and enable users to filter through a sea of stale content Game-ification Provide content producers with a sense of value Alerts& recommendations Move from active exploration to deliver relevant updates and content suggestions
- 129. The team
- 130. The Dataportal team Analytics&Experimentation Products John Bodley Software Engineer Eli Brumbaugh Experience Designer Jeff Feng Product Manager Michelle Thomas Software Engineer Chris Williams Data Visualization
- 131. The Dataportal team Analytics&Experimentation Products John Bodley Software Engineer Eli Brumbaugh Experience Designer Jeff Feng Product Manager Michelle Thomas Software Engineer Chris Williams Data Visualization