Enterprise Ready: A Look at Neo4j in Production
2 likes•1,340 views
The document outlines an agenda for a Neo4j training session held on April 26, 2017, in Santa Clara, emphasizing the advantages of graph databases. It discusses Neo4j's architecture, its application in real-time scenarios, and key benefits such as query productivity and system agility. It also highlights best practices for implementing Neo4j successfully within enterprise environments.
![27
Example HR Query in SQL The Same Query using Cypher
MATCH (boss)-[:MANAGES*0..3]->(sub),
(sub)-[:MANAGES*1..3]->(report)
WHERE boss.name = “John Doe”
RETURN sub.name AS Subordinate,
count(report) AS Total
Project Impact
Less time writing queries
• More time understanding the answers
• Leaving time to ask the next question
Less time debugging queries:
• More time writing the next piece of code
• Improved quality of overall code base
Code that’s easier to read:
• Faster ramp-up for new project members
• Improved maintainability & troubleshooting
Benefit #3 of 3: Query Productivity](https://image.slidesharecdn.com/neo4j-in-production-graphdays-santa-clara-2017-april-26-170428203321/85/Enterprise-Ready-A-Look-at-Neo4j-in-Production-27-320.jpg)
![MATCH (:Person { name:“Dan”} ) -[:MARRIED_TO]-> (spouse)
MARRIED_TO
Dan Ann
NODE RELATIONSHIP TYPE
LABEL PROPERTY VARIABLE
Key Ingredient #2 of 3:
A Productive and Powerful Graph Query Language](https://image.slidesharecdn.com/neo4j-in-production-graphdays-santa-clara-2017-april-26-170428203321/85/Enterprise-Ready-A-Look-at-Neo4j-in-Production-30-320.jpg)
![How Causally Consistent Reads Work
App Server
Driver
3: Review
Profile 4: Create
an order
Async
Replication
Raft
Replication
1: Read
Product
Catalog
Core ServerCore ServerReplica Server
App Server
Driver
App Server
Driver
ENTERPRISE EDITION
2. Create
Account
5: Review
orders How it Works:
• Application chooses a consistency level
“Read Any” vs “Read your own writes”
• Cluster chooses appropriate members
Default optimizes for scalability
(i.e. read replica server for reads)
Causal Clustering Enables:
• Application-driven SLAs
• Optimizing for freshness vs. cost
• Tunability within an application
On an application & session basis
1: Read any replica | 2: Write à[Tx 101] | 3: RYOW*[Tx 101] | 4: Write à[Tx 102] | 5: RYOW [Tx 102]](https://image.slidesharecdn.com/neo4j-in-production-graphdays-santa-clara-2017-april-26-170428203321/85/Enterprise-Ready-A-Look-at-Neo4j-in-Production-44-320.jpg)
Enterprise Ready: A Look at Neo4j in Production
- 1. 09:00-09:30 09:30-10:15 10:15-11:00 11:00-11:30 11:30-12:30 12:30-13:30 13:30-17:00 Breakfast and Registration The Connected Data Imperative: Why Graphs Transform Your Data: A Worked Example Break Enterprise Ready: A Look at Neo4j in Production Lunch Hands-On Training Session Agenda APRIL 26, 2017 SANTA CLARA
- 2. Key Takeaways 1. Neo4j architecture basics… to help you match up Neo4j with the right technical problem 2. Some guidelines for success in production 3. Where Neo4j fits into your enterprise architecture
- 3. The Right Data Technology For the Right Job Part I:
- 8. Hordes of Data Hoardes of Data One Perspective on “Big Data”
- 9. Trending & Aggregation Finding Needles in Haystacks One Perspective on “Big Data”
- 15. Discrete Data Minimally connected data Other NoSQL Relational DBMS Neo4j Graph DB Connected Data Focused on Data Relationships DBMSs Another Perspective on “Big Data”
- 16. Graph Graph Database Five Key Sub-Patterns (Including SQL) RDBMS TabularAggregate Oriented (3) Key-Value, Column-Family, Document Database Source: Martin Fowler NoSQL Distilled Database Management Systems
- 17. Illustration by David Somerville based on the original by Hugh McLeod (@gapingvoid) Connectedness Latency & Freshness Batch- Precompute Real-Time Important Dimensions in Technology Selection
- 18. Illustration by David Somerville based on the original by Hugh McLeod (@gapingvoid) RDBMS & Aggregate- Oriented NoSQL Hadoop / MapReduce |<———————- Graph Database & ———————>| Graph Compute Engine A View of the Data Management Portfolio
- 19. Latency & Freshness Batch- Precompute Real-Time Connectedness Illustration by David Somerville based on the original by Hugh McLeod (@gapingvoid) Neo4j Solves Connected, Real-Time Problems
- 21. Recommendations based on activity from yesterday Overnight/Intermittent Loading and Calculations Results in lag between activity & knowledge response System-wide local pre-calculations are computationally inefficient Real-Time Writes & Writes Up-to-the-moment freshness “Just-in-time” processing most efficient for “local” queries Recommendations that reflects your latest activity Batch Processing Real-Time Processing
- 22. Discrete Data Minimally connected data Hadoop Other NoSQL Relational DBMS Graph Database Connected Data Focused on Data Relationships Architectures for Leveraging Connectedness Designed for Discrete Lookups & Aggregation Designed for Causality & Pattern-Based Queries Architecture tradeoffs: - Data Model Richness for Volume - Performant Insight Into Connections - Data Trustability (ACID) Architecture tradeoffs: - Aggregation performance for arbitrary hop performance - “Infinite scale” for large scale index- free relationship performance
- 23. Distinguishing Features of a Native Graph Database Part II:
- 25. 25 A unified view for ultimate agility • Easily understood • Easily evolved • Easy collaboration between business and IT #1 Benefit: Project Agility The Whiteboard Model Is the Physical Model
- 26. Connectedness and Size of Data Set ResponseTime Relational and Other NoSQL Databases 0 to 2 hops 0 to 3 degrees Thousands of connections 1000x Advantage Tens to hundreds of hops Thousands of degrees Billions of connections Neo4j “Minutes to milliseconds” #2 Benefit: “Minutes to Milliseconds” Real-Time Query Performance
- 27. 27 Example HR Query in SQL The Same Query using Cypher MATCH (boss)-[:MANAGES*0..3]->(sub), (sub)-[:MANAGES*1..3]->(report) WHERE boss.name = “John Doe” RETURN sub.name AS Subordinate, count(report) AS Total Project Impact Less time writing queries • More time understanding the answers • Leaving time to ask the next question Less time debugging queries: • More time writing the next piece of code • Improved quality of overall code base Code that’s easier to read: • Faster ramp-up for new project members • Improved maintainability & troubleshooting Benefit #3 of 3: Query Productivity
- 29. At Write Time: data is connected as it is stored At Read Time: Lightning-fast retrieval of data and relationships via pointer chasing Index free adjacency Key Ingredient #1 of 3: Graph Optimized Memory & Storage
- 30. MATCH (:Person { name:“Dan”} ) -[:MARRIED_TO]-> (spouse) MARRIED_TO Dan Ann NODE RELATIONSHIP TYPE LABEL PROPERTY VARIABLE Key Ingredient #2 of 3: A Productive and Powerful Graph Query Language
- 32. “Why Neo4j”: What We Hear From Users ACID Transactions • ACID transactions with causal consistency • Neo4j Security Foundation delivers enterprise-class security and control Performance • Index-free adjacency delivers millions of hops per second • In-memory pointer chasing for fast query results Agility • Native property graph model • Modify schema as business changes without disrupting existing data Developer Productivity • Easy to learn, declarative openCypher graph query language • Procedural language extensions • Open library of procedures and functions APOC • Neo4j support and training • Worldwide developer community … all backed by Neo’s track record of leadership and product roadmap Hardware Efficiency • Native graph query processing and storage requires 10x less hardware • Index-free adjacency requires 10x less CPU
- 33. Recipes for Success with Neo4j Part III:
- 34. Confidential - Neo Technology, Inc. #1: Get to know the “Whole Product” Cloud IaaS, PaaSm, DBaaS Marketplace Companion Service Education Documents Online Training Classroom Custom Onsite 34 OSS Community Foundations LDBS, openCypher Events Forums Add-Ons Tech Ecosystem Tech Partners Graph Solutions Data Science Architecture Data Models Partners System Integrators Trainers OEMs Commercial Support Technical Support Packaged Services Custom Services
- 35. #2: Don’t be afraid to ask for help
- 36. #3: Use the technology for what it’s good for …not as a stashing ground for all data OLTP Relationships in Data Concrete Use Case
- 37. #4: Use the various APIs and components to your advantage Procedures - More complex imperative code - Extreme performance - APOC is your friend Cypher - Filtering & Pattern Matching - Most convenient Querying Bulk “neo4j-admin import” (1M rec/sec) Transactional - LOAD CSV - Community adapters & procedures - Roll-your-own Importing Data Community Edition - Learning - Simple projects Enterprise Edition - 24x7 - Large scale - Secure Product Edition
- 38. Deploying for 24x7 An Overview of Key Enterprise Features Part IV:
- 39. Real-time Package Routing • Large postal service with over 500k employees • Neo4j routes 7M+ packages daily at peak, with peaks of 5,000+ routing operations per second. Real-time promotion recommendations • Record “Cyber Monday” sales • About 35M daily transactions • Each transaction is 3-22 hops • Queries executed in 4ms or less • Replaced IBM Websphere commerce Real-time pricing engine • 300M pricing operations per day • 10x transaction throughput on half the hardware compared to Oracle • Presentation at http://graphconnect.com/gc2016-sf/ • Replaced Oracle database What’s Possible
- 40. Neo4j 3.1 Security and Clustering Architecture Build and deploy graph applications across an entire enterprise • Compliance with internal and external enterprise Information Security needs • Robust and flexible new clustering architecture for diverse operational scenarios and application needs A foundation that enables mainstream enterprise solutions on-premises and in the cloud ENTERPRISE GRAPH FOUNDATION Operational, Analytic, and Transactional Uses Security Clustering Operability Enterprise Graph Applications 40 The Graph Foundation for the Enterprise
- 41. Raft-based architecture • Continuously available • Consensus commits • Third-generation cluster architecture Cluster-aware stack • Seamless integration among drivers, Bolt protocol and cluster • Eliminates need for external load balancer • Stateful, cluster-aware sessions with encrypted connections Streamlined development • Relieves developers from complex infrastructure concerns • Faster and easier to develop distributed graph applications Neo4j Causal Clustering Architecture Resilient, Modern, Fault-Tolerant. Guarantees Graph Safety. 41 ENTERPRISE EDITION
- 42. How Causal Clustering Works 42 Replica Servers Query, View Core Servers Synced Cluster Read Replica Read- Write Read Replica Read- WriteRead Replica Read Replica Reporting and Analysis Graph App Driver BOLT Write Read Read Replica Read Replica Read Replica Built-in load balancing • Spreads reads to core and replica servers • Spreads writes across core servers Causal consistency • Always-consistent view of data at any scale • Stronger than eventual consistency • Supports varying app SLAs • Best model for graph transactions Large heterogeneous clusters • 1000+ instance clusters • No dependence on master avoids bottleneck • Mix and match instance types App servers, reporting servers, IoT devices… ENTERPRISE EDITION
- 43. R E P L I C A Q U E R I E S C O R E Q U E R I E S Causal Clustering Architecture Optimizes for Cost-Consistency at Query Time Read Any 43 Read Your Own Writes Read Any Read Your Own Writes Linearizable (Future 3.x) QUORATE The Holy Grail of Distributed Systems Q U E R Y C O S T ENTERPRISE EDITION
- 44. How Causally Consistent Reads Work App Server Driver 3: Review Profile 4: Create an order Async Replication Raft Replication 1: Read Product Catalog Core ServerCore ServerReplica Server App Server Driver App Server Driver ENTERPRISE EDITION 2. Create Account 5: Review orders How it Works: • Application chooses a consistency level “Read Any” vs “Read your own writes” • Cluster chooses appropriate members Default optimizes for scalability (i.e. read replica server for reads) Causal Clustering Enables: • Application-driven SLAs • Optimizing for freshness vs. cost • Tunability within an application On an application & session basis 1: Read any replica | 2: Write à[Tx 101] | 3: RYOW*[Tx 101] | 4: Write à[Tx 102] | 5: RYOW [Tx 102]
- 45. Consistency with Causal Clustering 45 Expected Consistency Behavior Eventual Consistency Neo4j Causal Consistency Every single server is eventually updated ✔ ✔ View of related data is always consistent ✔ Users reading and re-reading data always see the same data Unless there have been intervening updates by others ✔ Users writing and updating data always see the latest data Unless there have been intervening updates by others ✔ Eventual consistency is not good enough for graphs ENTERPRISE EDITION
- 46. Satisfy enterprise admin and database security requirements • Flexible authentication options ActiveDirectory/LDAP or Native users • Role-based Authorization • List and kill running queries • Access controls for User-Defined Procedures Enables subgraph access control • Query logging and Security event logging Passes through originating end user • Extendable Auth plugin Architecture Kerberos support coming soon! 46 Enables Sarbanes-Oxley, HIPAA, PCI-DSS, et al Neo4j Security Foundation Enterprise-Class Security and Control P R E D E F I N E D R O L E S Privileges Reader Publisher Architect Admin Change own password • • • • Read data • • • • View own details • • • • Terminate own query • • • • Write/update/delete data • • • Manage index/constraints • • Terminate others’ queries • ENTERPRISE EDITION
- 47. Neo4j Deployment Success Program End-to-end Neo4j support throughout the project lifecycle • Tailored expert advice to guide you all the way through to deployment • Ensures you are successful with Neo4j Dedicated Neo4j Expert • Design & Product Manager advice to avoid common mistakes • Topology advice to get you to production • Provide expert best practice guidance Deployment Success Engagement • Proactive solution review throughout the project’s lifetime • Continuous delivery of knowledge as we progress Sustained Customer Success
- 49. Coming Soon! Neo4j 3.2 • Multi Data Center • Even Faster Reads & Writes • More Schema Constraints • Add-on for Kerberos • Query Monitoring Improvements • And more…!
- 51. 09:00-09:30 09:30-10:15 10:15-11:00 11:00-11:30 11:30-12:30 12:30-13:30 13:30-17:00 Breakfast and Registration The Connected Data Imperative: Why Graphs Transform Your Data: A Worked Example Break Enterprise Ready: A Look at Neo4j in Production Lunch Hands-On Training Session Agenda APRIL 26, 2017 SANTA CLARA
- 52. Some Perspective We are still here Journeying to here
- 53. THE PROPERTY GRAPH DATA MODEL
- 54. A way of representing data DATA DATA
- 55. Relational Database Good for: • Well-understood data structures that don’t change too frequently A way of representing data • Known problems involving discrete parts of the data, or minimal connectivity DATA
- 56. Graph Database Relational Database A way of representing data Good for: • Dynamic systems: where the data topology is difficult to predict • Dynamic requirements: the evolve with the business • Problems where the relationships in data contribute meaning & value Good for: • Well-understood data structures that don’t change too frequently • Known problems involving discrete parts of the data, or minimal connectivity
- 58. Graph is easy to learn, hard to master • Common issues your team will hit • Underestimate graph complexity • Complaints of slow queries • Undersized hardware, especially memory, but also CPU • Ambitious number of future nodes • Bad scaling topology / architecture assumptions • Disappointing ‘Write’ speed • Deep analytics mismatch • You still need your 10,000 hours • 8760 hours in a year, so depending on how long you sleep, 5-7 years.
- 60. Enterprise Scale for Global Internet Applications • Causal Clustering works across data centers • Cores can be spread out across DCs • 1 leader, all followers, consensus commits • Read-follows-writes still ensured • Subclusters for speedier local activity • Replica can hierarchically map to local replicas • Cluster API-level control for developers • Cloud delivery via Azure and AWS EC2 Reston Data Center UK Data Center
- 61. Native Performance Improvements • Label Indexes added to speed inserts, updates and deletes • Compound indexes to improve operational speeds • Cypher’s depth query in “DISTINCT” function has been dramatically improved by eliminating repetitious traversals through deep levels. • Common Cypher queries can be compiled to improve performance • Improved performance of Neo4j browser with new JavaScript framework
- 62. Production Governance Improvements Neo4j is “Enterprise-Obedient” • Node Keys are now available as schema constraint • now specify keys for any label • This helps assure the integrity of your graph by enforcing existence and guaranteeing uniqueness • especially useful for applications exchanging or importing data from across multiple data sources. • Kerberos encrypted authentication module add-on • Supports 3-tier integration with client, directory and database • CAPI-Flash hardware from IBM Power8 add-on • Role-based control of queries in Query Monitor
- 63. HDFS/MapReduce/Spark (Storage & Aggregation) Streaming (Filtering & Aggregation) Machine LearningGraph Computation Software for “Big Data”
- 64. Interoperability
- 65. Extensible