![neo4j + Apache Spark + Gimel + GraphQL
Multiple Sources Systems
• Abstracts Graph Complexity
• Supports - Multiple Front-end
clients
• Accelerates UI Integration
• Reference | neo4j & GraphQL
• Spring-Data-neo4j [Scale]
Unified Access PatternData Processing
• Distributed Processing -
SCALE
• Spark-ML + Graphx. -
COMPLEXITY
• GIMEL + Spark - Data
access simplified
• Kafka Streams – Near Real
Time CDC in the future
Connected Components
• Easy Bootstrap
• Cypher & APOCs
• Robust libraries - Spark,
GraphQL
• Rich Graph Algorithms](https://image.slidesharecdn.com/graphofenterprisemetadata1-201020193634/85/Nodes2020-Graph-of-enterprise_metadata-NEO4J-Conference-17-320.jpg)
![GraphQL Query
GraphQL Query - Response
type column {
column:String!
storage_system_name:String!
dataset:String!
}
type dataset {
dataset:String!
storage_system_name:String!
column: [column]
@relation(name:"DATASET_HAS_COLUMN",
direction:OUT)
}
type Enterprise {
name:String!
}
Define GraphQL Schema
Cypher -> GraphQL
Cypher](https://image.slidesharecdn.com/graphofenterprisemetadata1-201020193634/85/Nodes2020-Graph-of-enterprise_metadata-NEO4J-Conference-24-320.jpg)
Nodes2020 | Graph of enterprise_metadata | NEO4J Conference
- 1. Graph of Enterprise Metadata– Powered by Neo4J & Spark Vladimir Bacvanski Deepak Chandramouli
- 3. AGENDA Prelude Introduction Architecture & Flow Scaling for enterprise Future Questions
- 4. About us Vladimir Bacvanski [email protected] Twitter: @OnSoftware • Principal Architect, Strategic Architecture at PayPal • In previous life: CTO of a development and consulting firm • PhD in Computer Science from RWTH Aachen, Germany • O’Reilly author: Courses on Big Data, Kafka Deepak Chandramouli [email protected] LinkedIn: @deepakmc • MT2 Software Engineer, Data Platform Services at PayPal • Data Enthusiast • Tech lead • Gimel (Big Data Framework for Apache Spark) • Unified Data Catalog – PayPal’s Enterprise Data Catalog
- 5. Prelude
- 6. 2019 | Recommendation System – Unified Data Catalog Building Recommendations for a Data Catalog A B S T R A C T I O N L A Y E R Yeah, let’s expand on this ! Graph of metadata has much more to Offer !!
- 7. Graph of Enterprise Metadata Introduction
- 8. The Enterprise Landscape 8 On-Premise Adjacencies Data Technologies Cloud Services Multi-DC / Hybrid Cloud Org Structures CollaborationPeople Identity & Access Security Zones Data Catalog Security Organization Artifacts Apps & Processes Logs Streams ETL Apps
- 9. Connecting the Dots… Effective Data Governance Privacy Regulation Complianc e …… The whole is greater ! Efficiency. Connected Components Security Organizatio n Data Business MetadataTechnical Metadata Artifacts Code Operational Metadata Internal Social Metadata Data Classification (PII/PCI/PHI)
- 10. Connected components = Graph of Enterprise Metadata
- 11. So What Top Down View ! ©2016 PayPal Inc. Confidential and proprietary.
- 12. Demo & Sandbox Code
- 13. Code Base | Playground Github https://github.com/Dee-Pac/GEM Gitter https://gitter.im/graph_of_enterprise_metadata/GEM-Ask-Us- Anything
- 14. Graph Build Architecture & Data Flow
- 15. Metadata Data Sources... • Data Classification • PII / PCI / PHI • Operational Metadata • Jobs & Apps • Databases & App – Query Logs • Org Structure • People, Geo , Hierarchies • Technical Data Catalog • Data Stores • Databases / Containers • Tables / Objects • Columns / Fields • Business Metadata / Glossary / Annotation • IAM • Access controls • Owners, Users • Social Metadata (Internal) • Slack Chats, Threads, Annotations • Artifacts • Code base – GitHub/SQL • Wiki – Documents, Mentions • JIRA – Issues, Mentions
- 16. High Level – Data Flow Merge Graph CRUD & Insights App Data Sources Data Sourcing Transform Application s Access Simplification Distributed Processing for scale (Future) Near Real Time - CDC Graph Access API Push
- 17. neo4j + Apache Spark + Gimel + GraphQL Multiple Sources Systems • Abstracts Graph Complexity • Supports - Multiple Front-end clients • Accelerates UI Integration • Reference | neo4j & GraphQL • Spring-Data-neo4j [Scale] Unified Access PatternData Processing • Distributed Processing - SCALE • Spark-ML + Graphx. - COMPLEXITY • GIMEL + Spark - Data access simplified • Kafka Streams – Near Real Time CDC in the future Connected Components • Easy Bootstrap • Cypher & APOCs • Robust libraries - Spark, GraphQL • Rich Graph Algorithms
- 18. <<<<<< Data Processing @ Scale >>>>>
- 19. © 2020 PayPal Inc. Confidential and proprietary. Technical Metadata >1000s of Data Stores >100K Containers >7 Million Datasets >50 Million Fields >10K PII/PCI Class elements Operational Metadata >100K Data Apps Per Day >25 Data Tech Stacks >API, Streams & Batch Apps Artifacts & Social Metadata, Artifacts >10K Scripts & App code >1000s of Wiki, JIRA Annotations Data Social ~ 12 Active Analytics Channel in Slack ~ Evolving Teams conversations for data Security & Organization >1000’s of IAM roles >20K Internal Users & Role Owners Metadata Growth
- 20. Gimel + spark + neo4j ©2016 PayPal Inc. Confidential and proprietary. GIMEL Data API NEO4J Connector
- 21. Gimel + spark + neo4j (Continued..) Unified Data API Unified Connector Config
- 22. Unified Data API Gimel + spark + neo4j (Continued..) Nodes & Relationship
- 23. Exposing Graph to Clients
- 24. GraphQL Query GraphQL Query - Response type column { column:String! storage_system_name:String! dataset:String! } type dataset { dataset:String! storage_system_name:String! column: [column] @relation(name:"DATASET_HAS_COLUMN", direction:OUT) } type Enterprise { name:String! } Define GraphQL Schema Cypher -> GraphQL Cypher
- 25. Conclusion & Next Steps
- 26. Key Take-aways • The whole is greater than Sum ! • Data Awareness: Data can be anywhere across the ecosystem : but we should know its shape & form of existence. • Graphs are everywhere: Full potential can be realized by connecting the dots - across board. • Enriching metadata: It is a collective effort; silos rarely achieve effectiveness. • Graph’s Effectiveness: expose the data in a consumable way for – all tech stacks alike !
- 27. What’s next? • Data Classification • PII / PCI / PHI • Org Structure • People, Geo , Hierarchies • Technical Data Catalog • Data Stores • Databases / Containers • Tables / Objects • Columns / Fields • Business Metadata / Glossary / Annotation • IAM • Access controls • Owners, Users • Artifacts • Code base – GitHub/SQL • Wiki – Documents, Mentions • JIRA – Issues, Mentions • Operational Metadata • Jobs & Apps • Databases & App – Query Logs • Lineage • Social Metadata (Internal) • Slack Chats, Threads, Annotations Connecting Further… Scale – Millions of Nodes X Edges Access - Simplification Graph – Data Security & Multitenancy Insights driven by – Graph / ML Algorithms Foundational Capabilities…
- 28. Questions? Code base – GITHUB_FOR_GEM Let’s collaborate - GITTER_FOR_GEM
- 29. Thank You!