[KubeConUS2019 Docker, Inc. Booth] Distributed Builds on Kubernetes with BuildKit and Docker Buildx
- 1. Tibor Vass, Docker Akihiro Suda, NTT Akihiro Suda, NTT Distributed Builds on Kubernetes with BuildKit and Docker BuildX 1
- 2. Why build images on Kube? 2 Two kinds of motivation 1. CI/CD 2. Developer Experience
- 3. About us 3 Why build images on Kube? 3 BK Pod BK Pod BK Pod Some Pod Some webhook 1. CI/CD
- 4. About us 4 Why build images on Kube? 4poor CPU, RAM, Wi-FI, battery 2. Developer Experience BK Pod BK Pod BK Pod Some Pod Some webhook 1. CI/CD 4
- 5. About us 5 Why build images on Kube? 5poor CPU, RAM, Wi-FI, battery 2. Developer Experience BK Pod BK Pod BK Pod Some Pod Some webhook 1. CI/CD 5 BuildX
- 6. Kubernetes driver for BuildX 6 $ docker buildx create --driver kubernetes --driver-opt replicas=3 --use $ docker buildx build -t foo --load . Set up BuildKit cluster on Kube automatically No YAML mess No TLS certificate mess
- 7. Kubernetes driver for BuildX 7 $ docker buildx create --driver kubernetes --driver-opt replicas=3 --use $ docker buildx build -t foo --load . Same user experience as legacy docker build But specify --load to load the result to local Docker (or --push to push)
- 8. Rootless mode 8 • You don’t want to run privileged pods? Don’t worry • BuildKit pods can be executed as a non-root user (UID 1000) without securityContext.privileged − Protect the host from potential BuildKit vulns − Friendly to PSP-restricted clusters − Fake root is emulated for Dockerfile RUNs docker buildx create … --driver-opt rootless=true
- 10. Caching 10 • Cache can be pushed to a registry as well as image BK Pod BK Pod BK Pod BuildX Registry Request Cache
- 11. Caching 11 • Cache can be pushed to a registry as well as image BK Pod BK Pod BK Pod BuildX Registry CacheRequest
- 12. Caching 12 • But remote cache might be slow compared to the BuildKit daemons’ local cache (/var/lib/buildkit in each of the Pods) • Example: − No cache: 2m50s − Remote cache: 36s − Daemon-local cache: 0.5s
- 13. Caching 13 Sticky mode allows sticking a build request to a specific Pod based on the Dockerfile path • Always hit cache • But non-optimal LB buildkitd-1 buildkitd-0 buildkitd-2 foo/Dockerfile bar/Dockerfile baz/Dockerfile --driver-opt loadbalance=(sticky|random) Circular hashing space
- 14. Parallelism 14 • Multiple Dockerfiles: built in parallel using multiple nodes • Multiple stages in a single Dockerfile: built in parallel using a single node − Future work: scatter stages across multiple nodes in parallel