Docker on docker leveraging kubernetes in docker ee
2 likes562 views
This document summarizes Docker's experience dogfooding Docker Enterprise Edition 2.0 which focuses on Kubernetes. Key aspects covered include planning the migration process over several months, preparing infrastructure to support both Swarm and Kubernetes workloads, upgrading control plane components, and migrating select internal applications to Kubernetes. Benefits realized include leveraging Kubernetes features like pods, cronjobs, and the ability to provide feedback that improved the product before general release.
Docker on docker leveraging kubernetes in docker ee
- 1. Brett Inman Senior Infrastructure Engineer, Docker Docker on Docker: Leveraging Kubernetes In Docker EE Manish Tomar Senior Software Engineer, Docker
- 2. v Using Docker EE at Docker Inc
- 3. ● We are a customer ○ SaaS and internal workloads ● Sharing what we’ve learned ○ Best practices, tips ● Dogfooding ○ Better product So?
- 4. ● Background ● Planning Process for EE2.0 ● Infrastructure Preparation ● Application Migration ● EE and Kube Features Agenda
- 5. ● EE - Docker Enterprise Edition ● UCP - Universal Control Plane ● DTR - Docker Trusted Registry ● SaaS - Software as a Service, e.g. Docker Hub Acronyms
- 6. Disclaimer
- 7. ● >200 hosts in production on EE ● >100 microservices ● Hub/Store/Cloud ○ 1 billion pulls every ~2 weeks ● Web properties (www, blog, etc) ● Build/CI systems Running on EE
- 8. Provide a self-service container platform. Role of Infrastructure
- 9. Interfaces for our users Role of Infrastructure ● Docker EE - Swarm and Kubernetes ● Logging ● Metrics and Monitoring
- 10. Use the platform to run applications and provide services. Role of Dev
- 11. ● Goal: provide real-world feedback pre-release ● Cross-functional ● Started in Summer 2016 ● EE 2.0 - focus on Kubernetes Dogfooding
- 12. v Planning for EE 2.0 / Kubernetes
- 13. Planning ● Milestones - assign owners and dates ● Weekly syncs ● Common slack channel #dogfooding ● Release blocker
- 14. Planning - milestones Create test cluster Deploy EE TP to test cluster Deploy swarm workloads to test cluster k8s routing setup Deploy k8s workloads to test cluster Deploy EE RC to staging Deploy k8s workloads to staging Deploy EE RC to production Soak test GA File issues
- 15. v Infra prep for EE2 and Kubernetes
- 16. ● Smaller clone of existing environments ● Make modifications to infra code ● PR changes back into stage/prod Test cluster
- 17. Sizing ● UCP Managers now running kube containers ● Minimum: 8GB ● Recommended: 16GB
- 18. Hosts per orchestrator Node Node Node Node Manager hub Node hub hub-k8s hub-k8s Node Manager Node Manager UCP Collection: /hub UCP Collection: /hub-k8s Kube namespace: hub-k8s
- 19. ● Calico ● Kubernetes concepts ○ Ingress Controllers ■ early, limited customization ○ Nodeport ■ simple building block Networking
- 21. ● Prometheus ○ every in-house application exposes a /metrics endpoint ○ exporters for third-party applications ● Service discovery ○ consul ○ kube api Metrics
- 22. scrape_configs: - job_name: 'kubernetes-pods' kubernetes_sd_configs: - role: pod api_server: https://ucp.{{ env "STACK" }}.domain.io:6443 bearer_token: {{ KUBE_TOKEN }} scheme: https Kubernetes service discovery in Prometheus Prometheus scrape config
- 23. Logging happens at the engine level - no change Logging
- 25. Upgrading UCP docker container run --rm -it --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp:3.0.1 upgrade --interactive
- 26. ● Images must be accessible on EVERY node ● No down nodes in UCP cluster ● Ubuntu 14.04 - see release notes for shared mounts ● Manager load balancing - kube API port 6443 ● Kube DNS Gotchas and Notes
- 27. Choosing applications Low Traffic Non-Critical High Traffic Non-Critical Low Traffic Critical High Traffic Critical
- 28. Choosing applications Low Traffic Non-Critical notifications
- 29. Choosing applications Low Traffic Non-Critical High Traffic Non-Critical registry2sns
- 30. ● No application code change ● No Dockerfile change ● No Environment changes What didn’t change
- 31. ● K8S yaml files add ○ Deployment and Service object ● Registry secrets ● All in Source control What did change
- 32. K8S deployment object kind: Deployment .. spec: ... template: metadata: labels: app: registry2sns spec: imagePullSecrets: - name: saasdeploycreds containers: - name: web image: docker/registry2sns:1093.0.0 ports: ...
- 33. K8S Service object apiVersion: v1 kind: Service metadata: name: registry2sns labels: app: registry2sns spec: type: NodePort ports: - port: 80 nodePort: 32769 selector: app: registry2sns
- 34. K8S Service object apiVersion: v1 kind: Service metadata: name: registry2sns labels: app: registry2sns spec: type: NodePort ports: - port: 80 nodePort: 32769 selector: app: registry2sns kind: Deployment .. spec: ... template: metadata: labels: app: registry2sns spec: imagePullSecrets: - name: saasdeploycreds containers: - name: web image: docker/registry2sns:1093.0.0 ports: ...
- 35. ● Test requests against k8s pods ● Service backed by swarm containers and k8s pods ● Change in deploy tooling Testing App Migration to K8S
- 36. v Leveraging EE and Kubernetes
- 37. Kubernetes api • kubectl • k8s api server • RBAC Docker api • Docker cli • Swarm • RBAC Web UI • Monitoring • Configuration • Single pane UCP Interfaces
- 38. Resource Sets Node Node Node Node Manager hub Node hub hub-k8s hub-k8s Node Manager Node Manager UCP Collection: /hub UCP Collection: /hub-k8s Kube namespace: hub-k8s
- 39. Airboss Based on engine labels: ● Set node orchestrator ● Create UCP collection ● Add node to UCP collection ● Apply collection label in kube ● Create kube namespace ● Create annotation linking collection label to namespace
- 40. Pods ● Deploy containers together ● Useful for ○ breaking up monoliths ■ localhost ■ sharing volumes ○ metric exporters
- 41. CronJobs ● Batch processing ● Moving system services into containers */5 * * * *
- 42. Dogfooding 10 P0/P1 fixes and improvements for EE2.0 162 total bugs and feature requests Takeaways
- 43. ● Planning - milestones, communication ● Infrastructure ○ Sizing ○ Resource sets ○ Routing ○ Metrics/Monitoring ○ Gotchas and Notes ● Migration process ● Leverage EE and Kube features Takeaways
- 44. v Go try it! Hosted Trial: trial.docker.com EE trial licenses: store.docker.com