Docker storage designing a platform for persistent data
5 likes3,047 views
This document summarizes a presentation about Docker storage and persistent data. It discusses immutable Docker images, applications with persistent data requirements, and different approaches to persistent storage with Docker including per-container storage, shared storage on the same host, multi-host shared storage, Docker volume plugins, and orchestrating storage with Swarm and Kubernetes. Key takeaways are that shared storage allows for smaller images, efficient usage of repetitive data, and decoupling of applications and data, and that running databases in containers is acceptable if requirements for latency, IOPS, bandwidth, and security are met.
![Volume plugin workflow
[dan@dockercon ~]$ docker volume –d array -o ssd -o 32Gb fast_volume
fast_volume
[dan@dockercon ~]$ docker volume ls
DRIVER VOLUME NAME
array fast_volume
[dan@dockercon ~]$ docker plugin install store/storagedriver/array](https://image.slidesharecdn.com/dockerstoragedesigningaplatformforpersistentdata-180620234956/85/Docker-storage-designing-a-platform-for-persistent-data-32-320.jpg)
Docker storage designing a platform for persistent data
- 1. Dan Finneran (@thebsdbox) EMEA Solutions Architect, Docker Docker Storage: Designing a Platform for Persistent Data
- 2. Why Storage?
- 3. Agenda What does immutable mean for your data Applications with Persistent Data requirements Persistent Data with Docker Docker volume plugins Orchestrating Storage (Swarm / Kubernetes) Key Takeaways / Conclusion Questions
- 4. Adjective: ”unchanging over time or unable to be changed: an immutable fact” Immutability
- 5. • Application + required libs/assets only • Designed to be automated • Re-Built ContainerOperating System • Kernel/Libs/Userland tools for all uses • Automation requires scripts /3rd party tools • Patched
- 6. Docker Image FROM alpine (Base image) COMMIT FE234B (Install binary packages) COMMIT 234CED (Copy assets or additional code) Dan's new Docker Image V1.0
- 7. Docker Container FROM dan/container:1.0 CoW layer (Copy on Write) $ docker run --rm dan/container:1.0 /test_file
- 8. Docker Container FROM dan/container:1.0 CoW layer (Copy on Write) $ docker run --rm dan/container:1.0
- 9. Applications with Persistent Data requirements
- 10. • Regardless of the lifespan of the container the data should always persist. • The container could be scheduled to run on any node in the cluster, meaning persistent data may need to be accessed from any node. Persistent Data
- 11. Accessing Data Block File API host01 / mnt iSCSI / Fibre Channel NFS REST
- 12. Is it wrong to run a Database in a container?
- 15. Latency IOPs Bandwidth / Throughput Security (external requirements) 21 34 Databases (+ additional requirements)
- 16. Batch processing Image Processing Format Conversion Transcoding Batch Processing • Watermark • Resizing • Formatting • Images • Documents • Custom data • Device handling • Bandwidth streaming • Tasks on multiple files/sources
- 17. A large number of applications will typically "park" cold data to a disk under the following circumstances: • Waiting for a back-end system to respond • Out of order data being processed • Data-sets that are typically too large to mapped to memory Applications that Require Persistent Data Between Restarts
- 18. Common bad practice patterns
- 19. Writing and storing logs inside a running container Push logs to an external / centralized platform
- 20. Package large datasets inside images Containers access datasets through shared storage
- 21. Persistent Data with Docker
- 22. •Per container storage •Shared storage (same host) •Multi-Host shared storage Persistence Implementations
- 23. Running your first Container (*) *This at least happened to me.
- 24. Per container storage / var lib docker volumes $ docker run –it -–rm dan/container:1.0 sh fc4b398edb01 host01 $ # Inside Container $ mkdir /storage / $ touch /storage/dockercon storage storage dockercon
- 25. Shared Storage (same host) / var lib docker volumes $ docker volume create dockercon dockercon $ docker run –it –-rm –v dockercon:/mnt busybox sh / mnt / mnt $ docker run –it –-rm –v dockercon:/mnt busybox sh host01
- 26. Host Persistence 192.168.0.2 host01 host02 host01 $ docker volume create --opt type=nfs --opt o=addr=192.168.0.2,rw --opt device=:/volume1/docker nfs nfs nfs
- 27. Host Persistence 192.168.0.2 host01 host02 nfs nfs / mnt host02 $ docker run –it –-rm –v nfs:/mnt busybox sh / mnt
- 30. • Extend the functionality of the Docker Engine • Use the extensible Docker plugin API • Allows an end-user to consume existing storage and its functionality • Create Docker storage volumes that are linked to a containers lifecycle (can be persisted afterwards if needed) Docker Volume Plugins
- 31. Current Plugins
- 32. Volume plugin workflow [dan@dockercon ~]$ docker volume –d array -o ssd -o 32Gb fast_volume fast_volume [dan@dockercon ~]$ docker volume ls DRIVER VOLUME NAME array fast_volume [dan@dockercon ~]$ docker plugin install store/storagedriver/array
- 33. Volume plugin workflow host01 API Call Mount volume/ mnt $ docker run –it –-rm –v dockercon:/mnt busybox sh
- 34. Data-intensive applications: Volume plugins expose specialized functionality in storage providers that can be utilised for data–intensive workloads. Database migration: Volume plugins make it easy to move data across hosts in the form of snapshots, which enable migration of production databases from one host to another with minimum downtime.. Plugin benefits / use-cases
- 35. Stateful application failover: Ability to have volumes that can be easily moved and re-attached, allowing easy failover to new machines/instances and re-attaching of data volumes. Reduced Mean Time Between Failures (MTBF): With volume plugins connected via a shared storage backend, operations teams can speed up cluster time-to-recovery by attaching a new database container to an existing data volume. This results in faster recovery of failed systems. Plugin benefits / use-cases
- 36. Orchestrating Storage with Docker EE 2.0
- 37. Storage with Swarm volumes: nfs: driver_opts: type: "nfs" o: "addr=docker01,nolock,soft,rw" device: ":/nfs" Swarm Volume As part of the compose file, specify a named volume and pass in the required settings Volume name
- 38. Storage with Swarm dockercon: image: dockercon:18 volumes: - type: volume source: nfs target: /nfs volume: nocopy: true Service Volumes In the service definition reference the volume along with additional configuration. Volume name Mount point
- 39. Storage with Kubernetes Persistent Volume A persistent volume when applied becomes a resource that is available to the cluster
- 40. Storage with Kubernetes Persistent Volume Claim A claim is a request from a user for a persistent volume, the request can include specifics such as: • Volume size • Volume capabilities • Access methods
- 41. Storage with Kubernetes Pod Defines a container or collection of containers that share networking and volumes. • Specify a PVC (persistent volume claim) • Map the PVC to a path inside the POD /http
- 42. Shared storage: • Smaller images • Efficient usage of repetitive data • Decouples application and data Key Takeaways
- 43. • Running a Database is fine in a container, as long as the requirements are met. • Don’t write logs inside a container Key Takeaways
- 44. Questions