Pods stuck "Terminating" on Windows nodes

[lizhuqi]

We're seeing pods that fail to terminate and have to manually run "kubectl delete --force" to force terminate the pods.

See the error message:
RemoveContainer "[ID]" from runtime service failed: rpc error: code = Unknown desc = failed to remove container "[ID]": Error response from daemon: unable to remove filesystem for [ID]: CreateFile C:\ProgramData\docker\containers[ID][ID]-json.log.4: Access is denied.

Similar issue was reported in AKS #1765 and fixed by upstream PR which is available for 1.17.12+. However the issues still repros in the 1.17.15.

[lizhuqi]

If you manually remove the json.log file, container can be removed and pod can then be ternimated.

[vrapolinario]

@lizhuqi can you please confirm this issue you're seeing is on AKS or is it outside of AKS? CC: @keikhara for visibility.

[lizhuqi]

It is seen in GKE.

[vrapolinario]

Got It. I added the label Windows on Kubernetes then. @immuzz or @keikhara can you please confirm this issue can be reproduced on AKS after the fix described above? If not, it might be something specific from the GKE platform.

[lizhuqi]

We took another look. It seems that the failure was caused by our logging agent handling symlinked log file and symlink commands result in access denied error in Windows containers. So I think the issue I reported here is a duplicate of #97.

[lizhuqi]

More logs for this issue.

2021-04-15T23:06:15ZHandler for DELETE /v1.40/containers/[CONTAINER-ID] returned error: unable to remove filesystem for [CONTAINER-ID]: CreateFile C:\ProgramData\docker\containers[CONTAINER-ID][CONTAINER-ID]-json.log: Access is denied.

2021-04-15T23:06:15.472679ZSyncLoop (PLEG): "[POD-NAME]_NAME", event: &pleg.PodLifecycleEvent{ID:"[ID]", Type:"ContainerDied", Data:"[DATA]"}

2021-04-15T23:06:15.472679Z[fake cpumanager] RemoveContainer (container id: [CONTAINER-ID])
2021-04-15T23:06:15.499033ZRemoveContainer "[CONTAINER-ID]" from runtime service failed: rpc error: code = Unknown desc = failed to remove container "[CONTAINER-ID]": Error response from daemon: unable to remove filesystem for [CONTAINER-ID]: CreateFile C:\ProgramData\docker\containers[CONTAINER-ID][CONTAINER-ID]-json.log: Access is denied.

2021-04-15T23:06:16.024224Zdetected rotation of /var/log/containers/[POD-NAME][NAMESPACE-NAME][CONTAINER-NAME]-[CONTAINER-ID].log; waiting 5 seconds

[lizhuqi]

Any update on this issue?

[lizhuqi]

Below is the docker version used:
Container Runtime Version: docker://19.3.13

[jsturtevant]

What OS version is your host? Can run cmd /c ver on the host?