Denodo Data Virtualization Platform: Scalability (session 3 from Architect to Architect webinar series)
4 likes2,575 views
The document outlines a webinar series on the Denodo platform, focusing on various aspects of data virtualization, including scalability, performance, and architectural overview. The session led by Jesus Barrasa highlights features such as high availability, load balancing, clustering, and the use of caching to optimize performance. Additionally, it discusses the Denodo Express, a free data virtualization solution, alongside strategies for disaster recovery and distributed environments.
Denodo Data Virtualization Platform: Scalability (session 3 from Architect to Architect webinar series)
- 1. Five In-depth Technology and Architecture Sessions on Data Virtualization Session 3: Scalability
- 2. Today’s Speaker ■ Jesus Barrasa Senior Solutions Architect, Denodo
- 3. Architect-to-Architect Series ■ Series of five webinars over 3 months ■ Deeper look into Denodo Platform ■ Architectural Overview ■ Performance ■ Scalability (today’s session) ■ Data Discovery and Governance ■ Security
- 4. Denodo Express ■ Denodo Express ■ Free to Download ■ Fully functioning Data Virtualization Platform ■ Single user, supports common data sources ■ Many of the same capabilities of Denodo Platform ■ Performance, Data Discovery, Governance, internal Security, Publishing, …
- 5. Scalability – Architecture Modules
- 6. Scalability – HA and Clustering ■ Denodo Cache ■ The Denodo Platform incorporates an advanced cache system to improve performance ■ For data materialization it makes use of an external JDBC database (physical or in-memory) ■ Denodo High-Availability ■ Set of features provided by the Denodo Platform to support High Availability and Load Balancing ■ Denodo Cluster Management ■ Tool for monitoring and managing a cluster of Denodo servers
- 7. Scalability – HA and Clustering ■ Singleton ■ High Availability Architecture ■ Active-Passive ■ Active-Active ■ Vertical Scalability ■ VM Support ■ Metadata Sharing ■ Horizontal Scalability ■ Clustering ■ Shared Cache or Individual Cache
- 8. • A single Data Virtualization server instance … Data sources LDAP Auth Cache Database Client Application ESB/BPM/Portal… Singleton
- 9. … Data sources LDAP Auth Cache Database Client Application ESB/BPM/Portal… HA – Active-Passive Load Balancer
- 10. HA – Active-Passive ■ All queries sent to Active server ■ Passive server running, but not handling requests ■ Both servers ‘pinged’ by Load Balancer with appliance health check or using ping script ■ When Load Balancer detects Active server failure ■ Requests redirected to Passive server ■ Passive server becomes ‘Active’ ■ When former Active server restored ■ Load Balancer determines new roles…
- 11. … Data sources LDAP Auth Cache Database Client Application ESB/BPM/Portal… HA – Active-Active Load Balancer
- 12. HA – Active-Active ■ Load Balancer distributes queries between both servers ■ Both servers running and handling requests ■ Both servers ‘pinged’ by Load Balancer with appliance health check or using ping script ■ When Load Balancer detects server failure ■ All requests redirected to other server ■ When ‘failed’ server restored ■ Added back to load balancing pool ■ Starts receiving queries
- 13. Vertical Scaling ■ Java limitations inhibit vertical scaling for singletons ■ Virtualization ■ Run Denodo Platform instances in VM ■ VMWare, Hyper-V, Xen, … ■ Very common scenario ■ Multiple Instances ■ Install multiple instances of Denodo Platform ■ Run multiple instances of Denodo Platform from a single Denodo Platform installation ■ Share metadata and cache
- 14. Vertical Scaling (Cont’d) Cache DatabaseMetadata Master Slave-1 Slave-2 Slave-3
- 15. Vertical Scaling (Cont’d) ■ First started instance is ‘master’ ■ All others are ‘slaves’ ■ Only ‘master’ instance can modify metadata ■ Slaves read metadata at startup time ■ Pros ■ Only one copy of metadata ■ Cons ■ Port configurations need modifying ■ SPoF
- 16. Horizontal Scaling – Clustering Load Balancer … Datasources Virtual Server VDP Cluster: 192.168.0.10:9999 Web Container Cluster: 192.168.0.10:9090 VDP Cluster: • Denodo1:9999 • Denodo2:9999 • Denodo3:9999 • Denodo4:9999 • Denodo5:9999 Web Cont. Cluster: • Denodo1:9090 • Denodo2:9090 • Denodo3:9090 • Denodo4:9090 • Denodo5:9090
- 17. Clustering – Shared Cache Load Balancer Virtual Server VDP Cluster: 192.168.0.10:9999 Web Container Cluster: 192.168.0.10:9090 VDP Cluster: • Denodo1:9999 • Denodo2:9999 • Denodo3:9999 • Denodo4:9999 • Denodo5:9999 Web Cont. Cluster: • Denodo1:9090 • Denodo2:9090 • Denodo3:9090 • Denodo4:9090 • Denodo5:9090 Cache Database
- 18. Clustering – Separate Caches Load Balancer Virtual Server VDP Cluster: 192.168.0.10:9999 Web Container Cluster: 192.168.0.10:9090 VDP Cluster: • Denodo1:9999 • Denodo2:9999 • Denodo3:9999 • Denodo4:9999 • Denodo5:9999 Web Cont. Cluster: • Denodo1:9090 • Denodo2:9090 • Denodo3:9090 • Denodo4:9090 • Denodo5:9090 Cache Database Cache Database Cache Database Cache Database Cache Database
- 19. Clustering - Caches ■ Shared Cache ■ Any server can populate cache ■ All servers can access cached data ■ Increased ‘hit ratio’ ■ Cache can become performance bottleneck ■ Separate Caches ■ Each server has its own cache ■ Avoids potential performance bottleneck ■ Access only data it has cached ■ Removes SPoF
- 20. Distributed Environments ■ Local caches help to achieve performance goals when servers are in geographically dispersed locations New York City Omaha San Francisco Houston
- 21. Distributed Environments (Cont’d) ■ A remote Denodo server can be viewed as a data source to another server ■ Promotes multi-layered architecture ■ Queries can be delegated to remote servers to reduce network data movement ■ Same query delegation mechanism as talked about in last session
- 22. Distributed Environments (Cont’d) ■ Remote query delegation… Local Data Sources Cache Cache Local Data Sources New York City HoustonClient 10 M rows 5 M rows JOIN Result: 20,000 rows JOIN Result: 2,500 rows 2 M rows
- 23. Distributed Environments (Cont’d) ■ Shared views… Local Data Sources Cache Cache Local Data Sources New York City HoustonClient 10 M rows 5 M rows JOIN Result: 20,000 rows JOIN Result: 2,500 rows 2 M rows New York City Houston
- 24. Disaster Recovery ■ Business Continuity policies usually require that critical IT systems are backed up to a DR center ■ Denodo supports scripted synchronization of metadata with DR systems ■ Using Export and Import tools ■ Sync up can be scheduled (time-based) or on an event (promotion to production)
- 25. Scalability - Summary ■ Denodo Platform scales from single instance to geographically distributed system ■ Grow system without reconfiguring ■ Supports HA configurations ■ Active-Passive, Active-Active ■ Vertical scalability options ■ Support for DR site synchronization
- 26. Q&A
- 27. Data Virtualization – Next Steps Move forward at your own pace Download Denodo Express – The fastest way to Data Virtualization Denodo Community: Documents, Videos, Tutorials, and more. Attend Architect-to-Architect Series Performance Scalability Move forward with one of our Data Virtualization experts Phone: (+1) 877-556-2531 (NA) Phone: (+44) (0)20 7869 8053 (EMEA) Email: [email protected] | www.denodo.com Data Discovery and Governance Security
- 28. Five In-depth Technology and Architecture Sessions on Data Virtualization Thank You! Next Session Session 4 Denodo Platform: Data Discovery and Governance