Database Containerization Platform Checklist
5 likes•501 views
The document outlines the checklist for a database containerization platform proposed by Robin Systems, addressing data management challenges and advantages of containerization over traditional virtual machines and bare-metal setups. It emphasizes the need for application-aware infrastructure, data persistence management, and quality of service to optimize performance and cost. Additionally, it presents various features such as cloning, snapshotting, and ease of management within container environments to enhance operational efficiency and agility.
Database Containerization Platform Checklist
- 2. Agenda › Data Management Challenges › Why Containers? › The Feature Checklist › Robin Solution and Demos › Summary
- 3. Data Management Landscape and Challenges Traditional Modern Data Platforms § Cost Containment o High software license & support costs § DevOps Agility o Quick database clone, snapshot and time-travel § Performance Predictability o Unpredictable performance in consolidated environments § Operational Complexity o Painful distributed application deployment, scaling and ongoing management § High Hardware CAPEX o Cluster sprawl, Data Duplication § Enterprise Grade Robustness o Backup, restore, failover, security
- 5. Containers – WHAT? › Next Generation of Virtualization technology › OS-based, and delivers bare metal performance › Operationally very similar to Virtual Machines › Dedicated IP address, file systems, separate user space, etc. › Agile › Containers can be launched in seconds › Better hardware consolidation › 10x or more higher consolidation benefits › Eliminate OS duplication and application software sprawl 0 20000 40000 60000 80000 100000 120000 Random Reads Random Writes Random Mixed IOPS Bare Metal Container VM Source: IBM Research http://domino.research.ibm.com/library/cyberdig.nsf/papers/0929052195DD819C85257D2300681E 7B/$File/rc25482.pdf Containers – 2x Faster than VMs
- 7. Cattle vs Pets! › Pets are given names like pussinboots.acme.com › They are unique, lovingly hand raised and cared for › When they get ill, you nurse them back to health › Examples: Databases, big data, and other stateful apps › Cattle are given numbers like inst0042.acme.com › They are almost identical to other cattle › When they get ill, you get another one › Examples: All stateless applications Future application architectures should use Cattle, but Pets with strong configuration management are viable and still needed * Original slide from Cern regarding different service models
- 10. 1. Multiple Container Types (and maybe even VMs) • The Ngnix • Node.js • Postgres • Redis • MariaDB • Node.js • ElasticSearch • Wordpress • Node.js SYSTEM CONTAINERS • Each container runs an entire service stack (multiple applications per container) • Meant to be used as lightweight VM • File system neutral • Examples – LXC, OpenVZ, Solaris Zones • Supports in-place patch/upgrade & SSH access Great for Traditional Applications Ngnix Node.js Postgres Redis MariaDB Node.js ElasticSearch Wordpress Ngnix APPLICATION CONTAINERS • Each container runs a single application (single concern per container philosophy) • Meant to be used as application package • Containers use layered file systems (Overlayfs, AUFS, etc) • Examples – Docker, Rocket • Patch/Upgrade entails replacing container image Great for Modern Applications
- 14. 3. Data Persistence Management › Application = multiple containers › Storage allocated per container, and not per host or VM › Multiple Volumes/container › Block storage: format to xfs, ext4, or raw › Each container can bind to multiple volumes – HDD, SSD, NVMe › Rapid create/destroy › Milliseconds response time to create/delete to keep up w/ containers › Higher volume › 10-100x more containers being created than VMs › Disk repair, replication, compression, …
- 16. 4. Cloning Prod to Test/Dev PROD Test Master Downstream environments Dev & Test Storage Usage for one application › Full Clones of your production database › Time consuming › Results in Data Sprawl › Create Test master to avoid performance penalty › Need a way to create as many environments but not pay the storage penalty ?
- 17. 4. Snapshot, Clone, and Restore 17 Prod Cluster Full Cluster, Distributed Application Snapshots OS + DB software + DB configuration + data Instant Cluster Clone Prod Scale Test Env in Minutes Need to test a change? Corruption or data loss? 2x storage Long data copy Manual re-configuration Erase commit logs Erase data files Restore backup Restart cluster Lot more than just storage snapshots Prod Cluster Prod Cluster Prod Cluster Prod Cluster Prod Cluster Virtual Test Cluster Instant Restore
- 19. 5. Database Workload Types Data Warehousing (OLAP, Reporting, Predictive Analytics) OLTP (Highly transactional system) › Low Latency › Impacts user experience › SSDs superior to HDDs › High Bandwidth i.e. Throughput › (Throughput = IOPS x Block size) › Deal with large volumes of data › Multiple databases hitting the same disks › Sustained high IOPS & Predictable low latency Consolidated =
- 20. 5. Need for Quality of Service (QoS) › Noisy Neighbor problem › Containers give process isolation but not performance isolation › Predictable performance › Guaranteed IOPS for application / container › Scale up/down › Traditional databases scale up; modern distributed databases scale out › But how do we cater to transient spikes? › Containers only allow for cpu and memory, but what about IOPS? Compute Disk 1 Disk 2 Disk N Consolidation w/ containers Compute Disk 1 Disk 2 Disk N
- 22. 6. As-Is Failover Container Failover, Migration APPLICATION CLUSTERStateless failover policies will not NOT work for databases ✔Container rescheduled to a different host x Container gets a new IP x Loses data as the volume is not remounted on the new host Key Use Cases: › Automatic failover › Manual node evacuation for planned maintenance (e.g. OS or hardware upgrades) › Cluster rebalance 22
- 24. 7. Network Automation › Docker networking is early stage and diverse (bridge, host, overlay, n/w driver) › Applications have to choose the best option › Docker, by default, uses Linux Ethernet Bridges for basic routing. This has a lot of performance issues. Same with Overlay network. › Use the host network stack instead (-net=host) Databases network requirements: › Span multiple hosts i.e. clustered › Each container should have unique name & IP addresses › Purposeful DB names (appdb.us.acme.com) registered in DNS › Expose well known ports (MySQL: 3306, Oracle: 1521) › Minimal network performance impact
- 26. 8. Multi-Tenancy & Isolation Tenant 1 Tenant 2 Tenant 2 Tenant 1 Hard Isolation Soft Isolation › Shared nothing; each tenant gets dedicated compute › Dedicated disk IO capacity › Add hardware for extra capacity › Poor consolidation due to silos › Shared compute, multi-tenancy provided by containers › Shared disk, but dedicated IO capacity › Containers resized for extra capacity › Higher consolidation
- 28. 9. Ease of Setup and Management 7 CLUSTER MANAGER (mesos, kubernates) 8 SCHEDULER (marathon, cronos) 2 STORAGE capacity tier (ceph, gluster,..) 4 NETWORKING (openvswitch, socketplane) 10 MONITORING (graphite, sensu) 5 CONFIGURATION (zookeeper, etcd) 9 UI (custom) 1 SERVER HARDWARE (supermicro, dell, …) 3 STORAGE performance tier (SSD, Flash, NVMe) 11 TROUBLESHOOTING (splunk, logstash) 6 CONTAINERS (docker, LXC) 1. When something breaks, hard to identify the root cause 2. If performance degrades, hard to identify hotspots 3. Extending the platform requires configuring multiple components 4. Keeping it operational requires knowledge of all components 11 Component Assembly
- 32. Platform for ALL Database & Big Data Applications Stateless Stateful Data Apps Enterprise Apps
- 33. Demos › Cluster Lifecycle Management › Initial deploy & scale out › Quality of Service › Dynamic & real-time control of CPU, Memory, & IOPS › Data Lifecycle Management › Snapshot, clone, and restore
- 34. › Reduce hardware footprint by 50% or more › Bare metal performance › Guaranteed Application-to-Spindle QoS Business Benefits Reduce CAPEX › Simplified application deployment › Self-service data lifecycle management Improve Productivity › 10x faster application deployment › Improved application uptime Reduce OPEX