Best Practices for Managing Docker Versions as presented at JavaOne 2017

@jbaruch #oraclecode
Docker Version –
Best Practices

About me
»@jbaruch
»Developer Advocate @JFrog
»We might still have some t-shirts left!

Poll Time!

Poll Time!
»Heard about Docker
»Can do the tutorial
»PoCing, playing etc.
»Production, baby!

JFrog Artifactory + Docker

»There are 2 hard problems in computer science:
cache invalidation,
naming things,
and off-by-1 errors.

Naming Things
with Docker

Who
Cares
About
Versions
Anyhow
?

Why Do We Care about versions?
»Pipeline Automation/Orchestration
»Traceability/Communication
»Maintainability/Storage Management

Those are old news,
we have been using versions for years
»Docker is different !!
»Versions are Tags

Docker Manifest and Tags
ac-image:1.0 ac-image:1.0-beta ac-image:1.0-qa
Manifest
sha256:252564..
Manifest
sha256:462564..
ac-image:1.1
OS: SHA2
Framework: SHA2
Application: SHA2: 3
Layers
Tags
Manife
st

So let’s talk about tags in detail
»Mutable and Dynamic in nature
»Example: my-image:5.0
»And Latest is not latest
»Confusion !!

my-image:5.0
OS layer
1.0
Framework
layer 2.0
Application
layer 2.0
OS layer
1.1
Framework
layer 2.1
Application
layer 2.1
Yesterday Today

The case for `latest`
And other mutable tags

Convenient promotion

Automation:
Docker Image Promotion Process across
silos

Promotion
ac-image:1.0-
dev
ac-image:1.0-qa
ac-image:1.0-
release
Manifest
sha256:252564..
OS: SHA2
Framework: SHA2
L
a
y
e
r
s
T
a
g
s

Pull, retag, push for every stage? It’s
nuts!!!

We got you covered

The case for immutable
tags

Traceability!
»We always know what this image is
»Trace it back to CI
»Trace it back to source

Traceability: Classic Approaches
»Version Names based On Git Hashes
»Version Names based on Jenkins Builds
»Version Names based on the packaged software
version number
»Version names with a unique date
(the build timestamp)

So which one is
better?

Static vs. Dynamic Tags
»Static Tags:
⋄Assigned at image CREATION
⋄Reflect metadata that traces the tag to its build and/or contents
»Dynamic Tags:
⋄Reflect an image you should use, the current known-good version
⋄Classic example is ubuntu:trusty
⋄Remember, ‘latest’ isn’t actually latest, just one tagged as such!

Love and Hate the dynamic versions
Pros Cons
Versions express promotion state Consistency
No version – trash! Traceability

Best Practices – Best of two worlds!

Double Tag
»Push every image with a static tag
»Create a second, dynamic tag for the same
image as appropriate
⋄Just a like a sym-link
⋄Tag name is not traceable by itself, because the “dynamic”
tag may have moved since client pulled.
⋄Docker Inspect, Search, Find image for traceability
»Cleanup static tags based on some policy

Static and Dynamic Tags
ac-image:1.0-
2017-04-01-
111
ac-image:latest1-
beta
ac-image:latest1
Manifest
sha256:252564..
Manifest
sha256:462564..
ac-image:1.0-
2017-04-03-
222
OS: SHA2
Framework: SHA2
L
a
y
e
r
s Application: SHA2: 4
T
a
g
s
ac-
image:latest1

The orphans

Another problem!
»So, if we change tags during promotion, what
happens to not promoted images?
»They loose their version!!!
(WTF, what does that mean?)

The unlucky ones
»In Java we call it SNAPSHOT-s.
»In Docker – images without versions.
»They are huge!

You don’t really have unlimited space

How Docker Registries
Work

Docker Distribution and DTR
»Checksum based storage
⋄Multiple tags with the same manifest does not use up additional disk
space
»Deleting a tag does NOT delete the layers from
storage
⋄Actual Delete based on SHA2 reference ONLY
⋄Tag is effectively deleted if you re-push with a same tag
⋄Old Manifest remains, and still referenceable by SHA2

Deleting a tag does NOT delete
the layers from storage
ac-image:1.0 ac-image:1.0-beta ac-image:1.0-qa
Manifest
sha256:252564..
Manifest
sha256:462564..
ac-image:1.1
OS: SHA2
Framework: SHA2
L
a
y
e
r
T
a
g
s

Actual DELETE is based on SHA2
reference ONLY
ac-image:1.0 ac-image:1.0-qa
Manifest
sha256:252564..
Manifest
sha256:462564..
ac-image:1.1
OS: SHA2
Framework: SHA2
L
a
y
e
r
T
a
g
s
Delete via
SHA2

Docker GC
»Required to delete layers with no manifests
»Required to clear up disk space
»Stop the world

Docker Distribution, Best Practice
»Delete layers without tags up front.
⋄HEAD Tag get SHA2 of manifest
⋄Delete SHA2
⋄After deleting reference, run GC to clean up

Docker Distribution, Best Practice
»(Not OOB) To clean up a registry with manifests
that have no tag
⋄Command used to find these layers:
comm -23 <(find . -type f -name "link" | grep
"_manifests/revisions/sha256" | grep -v "/signatures/sha256/" | awk -F/
'{print $(NF-1)}' | sort) <(for f in $(find . -type f -name "link" | grep
"_manifests/tags/.*/current/link"); do cat ${f} | sed 's/^sha256://g';
echo; done | sort) | wc –l

JFrog Artifactory
»We delete any layer that is not referenced by a
tag immediately
»We delete manifests that is not referenced by a
tag
»We have an API to copy/move a docker image or
change its tag without using the docker client to
pull/push a second time
»Configure the max number of tags per image

Built-in cleanup
ac-image:1.0 ac-image:1.0-qa
Manifest
sha256:252564..
Manifest
sha256:462564..
ac-image:1.1
OS: SHA2
Framework: SHA2
L
a
y
e
r
T
a
g
s
Delete via
SHA2
Manifest
sha256:462564..
ac-image:1.2
Delete Tag

Recap
»Dynamic versions are good for promotion
»Static versions are good for traceability
»Retagging should be done in the registry
»All unreferenced objects should be collected

Q&A and Links
»@jbaruch
»jfrog.com/shownotes
»We’re hiring!

Best Practices for Managing Docker Versions as presented at JavaOne 2017

More Related Content

Similar to Best Practices for Managing Docker Versions as presented at JavaOne 2017 (20)

More from Baruch Sadogursky (20)

Recently uploaded (20)

Best Practices for Managing Docker Versions as presented at JavaOne 2017

Editor's Notes