Windows platform matching logic does not allow for running HostProcess containers on any OS version when used in a manifest list

[marosset]

Description

Windows HostProcess containers do not inherit the same Windows OS version compactivity requirements as regular Windows Server containers. This means that a HostProcess container built can run on any Windows Server version.

The existing logic in the image platform matching code only returns a match when os=windows if the osversion matches what the currently running Windows Server version. This limits the useful ness of HostProcess containers by making them difficult to use in multi-arch manifest-lists.

Some use cases where this would be good to have working include Kubernetes infrastructure images like the kube-proxy, CNI images, etc. Since these workloads will need to run as HostProcess containers it would be desirable to include a single entry in the manifest-list for the Windows images (with arch=amd64 and os=windows specified but not having `osversion= set).

Using a single image for multiple Windows OS versions works today if the image is a single manifest and all platform attributes are omitted

Ex:

docker manifest inspect mrosse3/mcr-hpc-test:v0.1.0
{
        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
        "schemaVersion": 2,
        "config": {
                "mediaType": "application/vnd.docker.container.image.v1+json",
                "digest": "sha256:b01815998e5ac6bd7aed386a3c5dd94e9531b323a536dcf645f226a108966742",
                "size": 1345
        },
        "layers": [
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
                        "digest": "sha256:7dca6d47adc755dcbc146a8d26b251b7d08d5b8ce8336b04e1ad824cd3be763b",
                        "size": 20480
                },
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "digest": "sha256:0285fde2e7b1c2e110ff0e88966af55f1ef66f9e433591f5f613aba6aad831fb",
                        "size": 466
                }
        ]
}

Steps to reproduce the issue

1. Create a manifest-list containing one a Linux image and a Windows image (intended to be run as a HostProcess container)

docker manifest inspect mrosse3/hpc-manifest-list-test:latest
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 728,
         "digest": "sha256:081f486645e4e0bb51ae7fe455da6d6baa4c059b2a88ff7f4302de1b22d40d3c",
         "platform": {
            "architecture": "amd64",
            "os": "windows"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 526,
         "digest": "sha256:e2e85ebf25993e1528262fdcce76106aa4c2a9721fab5e944e1b93c0e3204a02",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      }
   ]
}

2. Schedule a Pod which runs this container to a Windows node

The result is the image fails to pull with the follow error

 Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  12s   default-scheduler  Successfully assigned default/ping to capz-conf-fg8px
  Normal   Pulling    11s   kubelet            Pulling image "mrosse3/hpc-manifest-list-test:latest"
  Warning  Failed     9s    kubelet            Failed to pull image "mrosse3/hpc-manifest-list-test:latest": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/mrosse3/hpc-manifest-list-test:latest": no match for platform in manifest: not found
  Normal   BackOff    9s    kubelet            Back-off pulling image "mrosse3/hpc-manifest-list-test:latest"

Describe the results you received and expected

I would expect this image to be able to be pulled onto the Windows node and the container to start.

I think there are a few different options for supporting this scenario.

• Update the platform matching logic for windows to be aware of some 'magic value' that indicates the image is a HostProcess container and we can skip matching on OSVersion
• Update the platform matching logic to allow for running images with only arch/os set on Windows (but still prefer to match n arch/os/version if there is a compatible image.

Note: If we decide to use a magic value we can update the Descriptor on the HostProcess container base image to set this value to ease container building them.

docker manifest inspect mcr.microsoft.com/oss/kubernetes/windows-host-process-containers-base-image:v1.0.0 -v
[
        {
                "Ref": "mcr.microsoft.com/oss/kubernetes/windows-host-process-containers-base-image:v1.0.0@sha256:4c5b2a2c0440d2fb50715774c5d1cb2a26997ce1d1d86586cbea882953bfad70",
                "Descriptor": {
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "digest": "sha256:4c5b2a2c0440d2fb50715774c5d1cb2a26997ce1d1d86586cbea882953bfad70",
                        "size": 420,
                        "platform": {
                                "architecture": "amd64",
                                "os": "windows"
                        }
                },
                "SchemaV2Manifest": {
                        "schemaVersion": 2,
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "config": {
                                "mediaType": "application/vnd.docker.container.image.v1+json",
                                "digest": "sha256:66db2536358359ac97453425532c82b15e9873f1630fbc94c6be80f75620f501",
                                "size": 870
                        },
                        "layers": [
                                {
                                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
                                        "digest": "sha256:f2784c8bc8525977aef9b6cd74e9a899274db7d82cd4b028e0ae049270927585",
                                        "size": 20480
                                }
                        ]
                }
        }
]

What version of containerd are you using?

v1.6.8 or latest built from main

Any other relevant information

No response

Show configuration if it is related to CRI plugin.

No response

[jterry75]

This is a similar issue to Hypervisor land as well. Basically the "runtime" affects the truth about the image pull/extract. The built-in matcher is the "Windows process container matcher" which only accounts for the rules of Argon's. We could add two more built-in matchers called "The Windows Host Container matcher" and the "Windows Xenon Matcher" I suppose. But it really does feel like only the caller has the right answers here. I assume since this is CRI we could pass the context required to know that this extraction is "going to be" a HostContainer and thus take different action?

[jterry75]

@TBBle - Is the expert here 🤣

[marosset]

This is a similar issue to Hypervisor land as well. Basically the "runtime" affects the truth about the image pull/extract. The built-in matcher is the "Windows process container matcher" which only accounts for the rules of Argon's. We could add two more built-in matchers called "The Windows Host Container matcher" and the "Windows Xenon Matcher" I suppose. But it really does feel like only the caller has the right answers here. I assume since this is CRI we could pass the context required to know that this extraction is "going to be" a HostContainer and thus take different action?

Oh I like this approach!
PullImageRequest contains PodSandboxConfig which does indicate if the pod is going to be running HostProcess containers and this could be used to pick a HostProcess contianer platform matcher.

PodSandboxConfig also contains the runtime handler name which could be used to help pick a different platform matcher for hypervisor isolated containers on Windows too in the future.

[TBBle]

PodSandboxConfig also contains the runtime handler name which could be used to help pick a different platform matcher for hypervisor isolated containers on Windows too in the future.

Yeah, that indeed sounds like the same sort of approach that was discussed and agreed as the best option for HyperV/Process isolation choice in #6491 (comment).

Reference is #6862 (comment) and #6862 (comment). I think the next step was #6657 making enough information available at the right layers that we could actually implement "chosen CRI runtime provides the matcher"; bits of that have landed since I was last able to look at this stuff, so it might be feasible to advance that part now, I'm not sure. The main open question I don't have an immediate resolution for there (as I'm still on sabbatical...) is how we turn an arbitrary runtime handler into a "Process or HyperV" decision at the platform-matcher-choice time, as CRI can't AFAIR see the Options["SandboxIsolation"] value or the PodAnnotations map, it just has the name of the runtime handler. (I think maybe the idea was that we'd have multiple snapshotter configs or something, each tied to its own matcher? That feels a little awful and repetitive, actually, so hopefully there's something nicer around.)

PullImageRequest contains PodSandboxConfig which does indicate if the pod is going to be running HostProcess containers and this could be used to pick a HostProcess contianer platform matcher.

Great, so PullImageRequest already knows if it's going to be HostProcess or not at the CRI layer. So for this ticket, I guess that could be implemented now, irrespective of the state of #6657 and the open questions in the above paragraph.

My thinking at the time was that the non-default PlatformMatchers (XenonMatcher and now HostProcessMatcher) would be implemented either directly in the CRI package, or as exported names from wherever the default matcher currently lives (sys/platform?) and either way, the CRI implementation (and ctr pull in parallel...) would be responsible for choosing a matcher to use. The logic of those matchers is pretty stable (approximately one change per Windows Server LTSC release, AFACT), so I don't see the need to try and make their behaviour particularly configurable, and hence would prefer they be in sys/platform for use by ctr and other API users, not just CRI. (Even in k8s-only cases, there's scope to improve container startup times with e.g., ctr pull --matcher HostProcess in the boot script of your k8s workload node)

There remains demand for overriding platform matchers to deliver site-local behaviour, e.g., #6693, presumably with some annotation on PullImageRequest naming a compiled-in-or-something custom matcher (or naming a local runtime hander which then refers to a compiled-in-or-something custom matcher), the actual design of which is out-of-scope here. So I guess all I'm saying in this paragraph is probably don't use a closed enum of (0=ProcessIsolationMatcher, 1=HyperVIsolationMatcher, 2=NoIsolationMatcher) for picking matchers.

Looking back on this, I notice it's a bit odd that HostProcess-ness is visible in CRI API, but every other isolation question (Process/Hyper-V/Kata/whatever) is not. I'm sure there's good reason for that (HostProcess containers are importantly different at the CRI level? WindowsSandboxSecurityContext suggests what the difference is...), but I assume that setting the HostProcess flag means that various flags in the runtime handler are either ignored, or configured runtime handlers are ignored completely. (I don't have the code at-hand to check this for myself, sorry.)

[TBBle]

Separate comment for new question:
Does this mean that HostProcess containers work by ignoring all the OS-type stuff in the container image (since they run in the host's... silo (?)), and just using the requested binary? So they wouldn't even need the OS-level stuff, and could just hold the relevant binary (and necessary non-OS dependencies like MSVCRT or similar, of course)?

If so, that'd make a nice new addition to the use-case list for microsoft/hcsshim#901, since that would allow HostProcess container images to be small and also provide another motivating case for FROM scratch in BuildKit WCOW support (assuming the multi-year project to land all the pieces for BuildKit WCOW support concludes successfully) on top of the fairly-nice use-case of microsoft/hcsshim#750

[marosset]

Today I made an attempt to solve this issue using platform matchers and ran into a problem.

I was able to create a new platform matcher to match on OS/arch and use the fields in the *runtime.PullImageRequest to add the desired platform matcher to the Pull call here

containerd/pkg/cri/server/image_pull.go

Line 163 in 740c933

image, err := c.client.Pull(pctx, ref, pullOpts...)

The issue I ran into is that after Pull there is a call to Update the image store and as far as I can tell there is not currently a way to specify a platform matcher for this call

containerd/pkg/cri/server/image_pull.go

Lines 186 to 188 in 740c933

	if err := c.imageStore.Update(ctx, r); err != nil {
	return nil, fmt.Errorf("failed to update image store %q: %w", r, err)
	}

This Update names creates an image object with the default platform matcher for the given OS

containerd/pkg/cri/store/image/image.go

Lines 73 to 89 in 740c933

	// Update updates cache for a reference.
	func (s *Store) Update(ctx context.Context, ref string) error {
	s.lock.Lock()
	defer s.lock.Unlock()
	i, err := s.client.GetImage(ctx, ref)
	if err != nil && !errdefs.IsNotFound(err) {
	return fmt.Errorf("get image from containerd: %w", err)
	}
	var img *Image
	if err == nil {
	img, err = getImage(ctx, i)
	if err != nil {
	return fmt.Errorf("get image info from containerd: %w", err)
	}
	}
	return s.update(ref, img)
	}

Which eventually results in the code not being able to find the image we just pulled resulting in a failure for the operation.

containerd/images/image.go

Lines 214 to 219 in 740c933

	var descs []ocispec.Descriptor
	for _, d := range idx.Manifests {
	if d.Platform == nil || platform.Match(*d.Platform) {
	descs = append(descs, d)
	}
	}

I'm pretty new to the containerd codebase so may be missing something obvious here.
@jterry75 @dcantah @TBBle do you have any ideas / suggestions?