![2014/10/30 21:35:31 Error getting container init rootfs
b528d54a0458a8cd8a798309930adb45cb5e1a7430e981e0f
3108f86386aab67 from driver devicemapper: open
/dev/mapper/docker-9:127-14024705-
b528d54a0458a8cd8a798309930adb45cb5e1a7430e981e0f
3108f86386aab67-init: no such file or directory
make: *** [build-django] Error 1
Build step 'Execute shell' marked build as failure
@bridgetkromhout
breaking builds](https://image.slidesharecdn.com/dockerinproduction-realitynothype-chicago-150310143914-conversion-gate01/85/Docker-in-Production-Reality-Not-Hype-DevOps-Chicago-18-320.jpg)
Docker in Production: Reality, Not Hype - DevOps Chicago
- 1. @bridgetkromhout Docker in Production Reality, Not Hype Bridget Kromhout
- 2. @bridgetkromhout Bridget Kromhout Operations Engineer @DramaFever Minneapolis, Minnesota @devopsdays @devopsdaysMSP @arresteddevops bridgetkromhout.com
- 3. @bridgetkromhout K-dramas since 2009. Docker in prod since October 2013.
- 4. @bridgetkromhout architecture ● All services in AWS ● Python (Django) main website ● Go microservices (video analytics ingest, on-the fly image processing, bookmarking…) ● Upstreams routed via nginx ● Celery + SQS for async tasks
- 5. @bridgetkromhout scale ● Streaming delivery through Akamai ● peak load at 10s of thousands of requests per second to the website ● The sine wave has a 10-20x differential throughout the week
- 6. @bridgetkromhout Why Docker? consistent development repeatable deployment
- 8. private registry: the official party line https://github.com/docker/docker- registry#quick-start S3 is a storage engine option but… a central registry server didn’t scale well for us @bridgetkromhout
- 9. private registry: dramafever @bridgetkromhout # this goes in /etc/default/docker to control docker's upstart config DOCKER_OPTS="--graph=/mnt/docker --insecure- registry=local-repo-alias.com:5000" ● local-repo-alias.com in DNS with A record to 127.0.0.1 ● OS X /etc/hosts: use the boot2docker host-only network IP
- 10. registry upstart docker pull public_registry_image docker run -p 5000:5000 --name registry -v /etc/docker-reg:/registry-conf -e DOCKER_REGISTRY_CONFIG=/registry-conf/config.yml public_registry_image @bridgetkromhout
- 11. config.yml s3_region: us-east-1 s3_access_key: <aws-accesskey> s3_secret_key: <aws-secretkey> s3_bucket: <bucketname> standalone: true @bridgetkromhout
- 12. what even is flate?! Pulling repository local-repo-alias.com:5000/www 4dda2b433370: Error pulling image (prod) from local-repo-alias.com:5000/www, flate: corrupt input before offset 54393671 flate: corrupt input before offset 54393671 d497ad3926c8: Error downloading dependent layers 2014/12/07 02:34:54 Error pulling image (prod) from local-repo-alias.com:5000/www, flate: corrupt input before offset 54393671 @bridgetkromhout
- 13. registry rewrite coming! DOCKER_OPTS="--graph=/mnt/docker -- insecure-registry=local-repo-alias. com:5000 -e STORAGE_REDIRECT=true" ...until we get to the promised go lan(d|g), there’s a workaround for the flate errors we’re seeing: @bridgetkromhout
- 16. weekly base builds FROM local-repo-alias.com:5000/www-base ● include infrequently-changing dependencies ○ ubuntu packages ○ pip requirements ○ wheels ● other builds can start from these images (so they’re faster): @bridgetkromhout
- 17. sudo docker build -t="a12fbdc" . sudo docker run -i -t -w /var/www -e DJANGO_TEST=1 -- name test.a12fbdc a12fbdc py.test -s sudo docker tag a12fbdc local-repo-alias.com: 5000/www:'dev' sudo docker push local-repo-alias.com:5000/www:'dev' @bridgetkromhout www-master build
- 18. 2014/10/30 21:35:31 Error getting container init rootfs b528d54a0458a8cd8a798309930adb45cb5e1a7430e981e0f 3108f86386aab67 from driver devicemapper: open /dev/mapper/docker-9:127-14024705- b528d54a0458a8cd8a798309930adb45cb5e1a7430e981e0f 3108f86386aab67-init: no such file or directory make: *** [build-django] Error 1 Build step 'Execute shell' marked build as failure @bridgetkromhout breaking builds
- 20. @bridgetkromhout Retry the build… ...only if a specific regex appears
- 21. @bridgetkromhout useful for unattended base builds need to change how it reports to Slack
- 22. @bridgetkromhout tag for staging tag for prod out of ELB restart upstart back in ELB Ship it!
- 24. before summer 2014 Vagrant for local development chef-solo provisioner 17 minutes to install everything @bridgetkromhout
- 25. now: boot2docker devs pull down images built on jenkins mysql image is built with fixtures can run master or qa image (or even prod) can build new local images from Dockerfiles @bridgetkromhout
- 26. local registry for dev docker run -d -p 5000:5000 --name docker-reg -v ${DFHOME}:${DFHOME} -e DOCKER_REGISTRY_CONFIG=${DFHOME} /config/docker-registry/config.yml public_registry_image @bridgetkromhout
- 27. $ boot2docker ssh date -u Mon Nov 24 16:09:02 UTC 2014 $ date -u Tue Nov 25 01:43:49 UTC 2014 @bridgetkromhout time is what turns kittens into cats
- 28. S3 requires clock sync $ docker pull local-repo-alias.com:5000/mysql Pulling repository local-repo-alias.com:5000/mysql 2014/11/24 19:44:31 HTTP code: 500 $ boot2docker ssh sudo date --set "$(env TZ=UTC date '+%F %H:%M:%S')" @bridgetkromhout
- 29. Devs can use their preferred editing environment: -v ${DFHOME}/www:/var/www We still want logs, too, so we expose those for the dev here: -v ${DFHOME}/www/run:/var/log volume mounting & our fork @bridgetkromhout Until 1.3 we ran a forked boot2docker We needed to mount local files into the VM
- 30. containerizing front-end useful for building front-end apps on Jenkins also allows consistent testing RUN apt-get install -y nodejs nodejs-legacy npm RUN npm install -g [email protected] RUN npm install -g [email protected] RUN npm install -g [email protected] ADD bower.json /var/www/dependencies/bower.json RUN cd /var/www/dependencies && bower install -- allow-root --config.interactive=false --force @bridgetkromhout
- 31. @bridgetkromhout django: image: local-repo-alias.com:5000/www:dev ports: - "8000:8000" links: - mysql - redis environment: - PYTHONPATH=/var/local - DJANGO_ENVIRON=LOCAL - DB_PORT_3306_TCP_ADDR=mysql command: /var/local/config/local/start-django-local volumes: - ${DFHOME}/www/run:/var/log - ${DFHOME}/www:/var/local mysql: image: local-repo-alias.com:5000/mysql:dev expose: - "3306:3306"
- 32. for persistent instances # remove stopped containers @daily docker rm `docker ps -aq` # remove images tagged "none" @daily docker rmi `sudo docker images | grep none | awk -F' +' '{print $3}'` @bridgetkromhout
- 33. failure modes cron zombies out of memory errors race conditions @bridgetkromhout
- 34. what isolation? -v /var/log/containers:/var/log @bridgetkromhout Host instances moving into
- 36. through a container darkly: monitoring @bridgetkromhout
- 37. containers building (lighter) containers easier with statically linked binaries go microservices android apk @bridgetkromhout
- 38. $ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE local-repo-alias.com:5000/mysql dev b0dc5885f767 2 days ago 905.9 MB local-repo-alias.com:5000/www dev 82cda604a4f1 2 days ago 1.092 GB local-repo-alias.com:5000/micro local bed20dc84ea1 4 days ago 10.08 MB google/golang 1.3 e3934c44b8e4 2 weeks ago 514.3 MB public_registry_image 0.6.9 11299d377a9e 6 months ago 454.5 MB scratch latest 511136ea3c5a 18 months ago 0 B $ ever-smaller images @bridgetkromhout