node GPU visible to pod container even when NO GPU resource limit is specified in pod spec

[EdwardZhang88]

1. Issue or feature description

When I submit K8S jobs, I find that even though I don't allocate any GPU resource in YAML spec, I can still see all the GPUs in the pod container if the image is built with CUDA support. Is this because by default nvidia-docker2 adds a hook to docker container startup, which makes all GPU devices visible to the container and GPU isolation will be applied only when GPU limit is explicitly specified in pod spec？I think this is a bug as user might accidentally get access to all GPU while they shouldn't.

2. Steps to reproduce the issue

1. Install nvidia-docker 2 and set default-runtime to nvidia
2. Deploy device-plugin daemonset
3. Create a YAML which has no GPU limit but uses a CUDA image
4. Submit the YAML and check inside the created container by nvidia-smi command
5. All GPU on the host node will be visible in the container while no GPU is expected

3. Information to attach (optional if deemed irrelevant)

I am using Docker CE 18.03 and Nvidia-docker 2.0.3.

[RenaudWasTaken]

Hello!

Yes this is a known limitation of the device plugin API and we are working with the community to fix it.

A workaround is to use pod presets to inject into all the pods the environment variable NVIDIA_VISIBLE_DEVICE = VOID

[zionwu]

Hello!

Yes this is a known limitation of the device plugin API and we are working with the community to fix it.

A workaround is to use pod presets to inject into all the pods the environment variable NVIDIA_VISIBLE_DEVICE = VOID

The env NVIDIA_VISIBLE_DEVICE is missing a 'S'. It should be 'NVIDIA_VISIBLE_DEVICES' according to the doc https://github.com/nvidia/nvidia-container-runtime#nvidia_visible_devices

[zw0610]

Is this issue really fixed by NVIDIA instead of a workaround?

[klueska]

This is the latest best practice to prevent this: https://docs.google.com/document/d/1zy0key-EL6JH50MZgwg96RPYxxXXnVUdxLZwGiyqLd8/edit