[CRI] unable to pull dockerhub library image, reference for unknown type: application/octet-stream

[BenTheElder]

Description

When running a pod with a certain image under containerd-cri the image pull fails. It appears this is due to

Nov 19 16:12:00 kind-control-plane containerd[136]: time="2020-11-19T16:12:00.140053300Z" level=warning msg="reference for unknown type: application/octet-stream"

Steps to reproduce the issue:

1. Create containerd based kubernetes cluster
   (I used git clone https://github.com/kubernetes-sigs/kind && cd kind && git checkout aebedcf8f4bf68a139b9572c0f18656cc1a9e429 && make build && bin/kind create cluster)
2. Create a pod using redis:2.8.23
   (kubectl apply -f https://raw.githubusercontent.com/kubernetes/examples/master/guestbook-go/redis-master-controller.json)
3. Observe failure to pull image
   (kubectl describe po)

Events:
  Type     Reason            Age                    From                         Message
  ----     ------            ----                   ----                         -------
  Warning  FailedScheduling  2m50s (x2 over 2m50s)  default-scheduler            0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
  Normal   Scheduled         2m30s                  default-scheduler            Successfully assigned default/redis-master-8qcns to kind-control-plane
  Normal   Pulling           2m29s                  kubelet, kind-control-plane  Pulling image "redis:2.8.23"
  Warning  Failed            2m26s                  kubelet, kind-control-plane  Failed to pull image "redis:2.8.23": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/redis:2.8.23": failed to unpack image on snapshotter overlayfs: failed to extract layer sha256:4dcab49015d47e8f300ec33400a02cebc7b54cadd09c37e49eccbc655279da90: failed to get reader from content store: content digest sha256:51f5c6a04d83efd2d45c5fd59537218924bc46705e3de6ffc8bc07b51481610b: not found
  Warning  Failed            2m26s                  kubelet, kind-control-plane  Error: ErrImagePull
  Normal   Pulled            11s (x11 over 2m26s)   kubelet, kind-control-plane  Container image "redis:2.8.23" already present on machine
  Warning  Failed            11s (x11 over 2m26s)   kubelet, kind-control-plane  Error: failed to create containerd container: error unpacking image: failed to extract layer sha256:4dcab49015d47e8f300ec33400a02cebc7b54cadd09c37e49eccbc655279da90: failed to get reader from content store: content digest sha256:51f5c6a04d83efd2d45c5fd59537218924bc46705e3de6ffc8bc07b51481610b: not found

4. Make sure the cluster is in a clean state
   (I used bin/kind delete cluster && bin/kind create cluster)
5. On the node, manually pull to the Kubernetes/CRI namespace
   (docker exec kind-control-plane ctr -n=k8s.io images pull docker.io/library/redis:2.8.23)
6. Create the pod again
   (kubectl apply -f https://raw.githubusercontent.com/kubernetes/examples/master/guestbook-go/redis-master-controller.json)
7. Observe success
   (kubectl describe po)

This is repeatable, and repros what @justinsb originally found on kops/aws with containerd 1.4.1.

What's especially odd here is how it succeeds under manual pull with ctr but not when leaving the pull up to CRI.

Describe the results you received:

Image failed to pull successfully under CRI.

Describe the results you expected:

Image pull should succeed.

Image should pull successfully (appears to work on docker).

Output of containerd --version:

containerd github.com/containerd/containerd v1.4.0-106-gce4439a8 ce4439a8151f77dc50adb655ab4852ee9c366589

Any other relevant information:

When doing the pull you will see a warning about a layer, in the containerd logs under CRI you see:

Nov 19 16:11:56 kind-control-plane containerd[136]: time="2020-11-19T16:11:56.975489200Z" level=info msg="PullImage \"redis:2.8.23\""
Nov 19 16:12:00 kind-control-plane containerd[136]: time="2020-11-19T16:12:00.140053300Z" level=warning msg="reference for unknown type: application/octet-stream" digest="sha256:481995377a044d40ca3358e4203fe95eca1d58b98a1d4c2d9cec51c0c4569613" mediatype=application/octet-stream size=5946

I see downstream users avoiding images with this issue, but sadly did not find an upstream bug so far.

AliyunContainerService/pouch#1583

[ktock]

This seems to be cased by the containerd client's Pull API logic rather than CRI.
redis:2.8.23 has a legacy application/octet-stream config blob (unsupported by Docker/OCI specs; see also #2456 (comment)), which isn't recognized by the Pull API logic so layer won't be pulled.

$ curl -H "Accept: application/vnd.docker.distribution.manifest.v2+json" ...omit... https://registry-1.docker.io/v2/library/redis/manifests/2.8.23 | jq '.config'
{
  "mediaType": "application/octet-stream",
  "size": 5946,
  "digest": "sha256:481995377a044d40ca3358e4203fe95eca1d58b98a1d4c2d9cec51c0c4569613"
}

The solution will be to just add application/octet-stream MediaType here. But I'm not sure we should support this MediaType.

containerd/unpacker.go

Line 319 in af963cc

case images.MediaTypeDockerSchema2Config, ocispec.MediaTypeImageConfig:

What's especially odd here is how it succeeds under manual pull with ctr but not when leaving the pull up to CRI.

ctr uses different API so it doesn't see this.
We can repro this manually with crictl.

# crictl pull docker.io/library/redis:2.8.23
...
FATA[0010] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/redis:2.8.23": failed to unpack image on snapshotter overlayfs: failed to extract layer sha256:4dcab49015d47e8f300ec33400a02cebc7b54cadd09c37e49eccbc655279da90: failed to get reader from content store: content digest sha256:51f5c6a04d83efd2d45c5fd59537218924bc46705e3de6ffc8bc07b51481610b: not found 

[justinsb]

Thanks for filing & investigating @BenTheElder & @ktock !

I agree we should update the redis image and I'll work on that - the whole example is very out of date. I don't know whether this is could be a problem for containerd/crictl (i.e. whether similar problems exist in other images); we couldn't find an existing bug for this problem.

[thaJeztah]

/cc @tianon

[estesp]

There was fairly complete discussion on this in #2456. I think the net of it is:

• images built prior to Docker 1.11/mid-2016 could have the wrong media type encoded in the config; the bug was fixed in April 2016; released in Docker 1.11.
• While I understand it might be possible to depend on an image that is now over four years old, that flies pretty hard in the face of "cloud native" best practices in 2020 :)

The open question in my mind (possibly directed at @tianon 😇 ): I assume these very old images are truly not "official images" (in the sense of updates/support) anymore--has there ever been any thought to actually cull them from DockerHub?

[tianon]

Those old images were official images, but you are correct that they are no longer supported -- here's a standard reply we give to maintainers that I think might be helpful context:

Removing tags here will remove them from the "Supported" section on the Hub readme (and will prevent us from spending cycles rebuilding them on the official build servers), but the tags will still be available to users who want them. (See https://github.com/docker-library/official-images#library-definition-files for more detail on this.)

(And just to be explicitly clear, official images have never been built any more esoterically than docker build + docker push at the lowest levels, so this being due to bugs in Docker 1.11 seems likely 😞)

[BenTheElder]

This issue cropped up because Kubernetes examples repo has very old sample deployments etc. that otherwise still function for demo purposes, so the images are not actively upgraded because nobody truly runs these. Of course we want them to work on containerd though.

Bumping seems reasonable, though I wonder what the impact of compat for this would be.

[BenTheElder]

To clarify:

• I think we clearly need someone in the Kubernetes space to patch up examples to supported images (as we've done for the particular image from this issue).
• I wonder if more users will be surprised when migrating dockershim => containerd breaks running some image. I think a surprising amount of ""cloud native"" users are not necessarily keeping everything on the bleeding edge 🙃 .. If the compatibility patch would be small (I suspect it is, but I'm not sure?), it might be worth reconsidering?

[estesp]

Yeah, it's a tough spot, but what makes me hope to hold the line here is:

• images with this "bug" should be extremely rare. If we were talking this is something that happened to a lot of images from 6 months or even a year ago, I get it. Given we are approaching year 5 (in a few months) since this was fixed, I'm thinking we can figure out how to do some better image hygiene, even in examples. I know it's a pain--I have lots of out of date code in my GitHub examples, too!
• we really should be in an environment in 2020 where, as much as possible, compliance to image spec should be held as a gold standard by all build tools and runtimes. Containerd has some experience with this re: registry/distribution implementations that depend on lax Docker behaviors, but in the end, we've convinced several registries to fix inconsistencies rather than dot the code with a lot of "oh, and we have to allow this because.." clauses. It's much harder to ever remove those, as you might imagine.

I hope this particular issue with dockershim -> containerd migration experience is extremely rare, and hopefully would not impact anyone who is building and deploying code today as they will never have images built with Docker 1.10, I would hope! I'm going to ping @containerd/containerd-maintainers as others might have a different view.

[thaJeztah]

I generally agree; I don't see a direct need for containerd to carry backward compatibility code for images created before containerd 1.0 even existed.

That said; I asked in our internal slack if we have data on manifests still being pushed to Docker Hub with this mediatype in the last year or so

[tianon]

That said; I asked in our internal slack if we have data on manifests still being pushed to Docker Hub with this mediatype in the last year or so

IMO the more interesting additional metric would be the frequency of users pulling them 😇

[kzys]

Resolving? containerd won't support container images with application/octet-stream.