-
Notifications
You must be signed in to change notification settings - Fork 3k
Description
Issue Description
When requesting GPUs in a container that uses UID mappings, the createContainer hooks that are required to perform certain setup operations in the container fail to run because they do not have permissions to read the container config.json from the bundlePath. The container config is required to determine the container root.
For example:
podman run --uidmap 0:1000000:65535 --gidmap 0:1000000:65535 --device nvidia.com/gpu=all ubuntu nvidia-smi
fails with:
Error: OCI runtime error: crun: {"msg":"error executing hook `/usr/bin/nvidia-cdi-hook` (exit code: 1)","level":"error","time":"2025-12-13T12:54:14.665626Z"}
As per NVIDIA/nvidia-container-toolkit#648, using runc as a container runtime shows more information for the error:
Error: /usr/bin/runc: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time="2024-08-13T09:49:32+02:00" level=error msg="failed to determined container root: failed to open OCI spec file: open /home/lahwaacz/.local/share/containers/storage/overlay-containers/981df8bef771d05f1e2f2a907cf6a0d0698ab2f24dedb34201ef14d91c6959cc/userdata/config.json: permission denied": OCI permission denied
Is this expected behaviour?
Since it was reported to have worked in Podman 4.9.4 but failing with Podman 5.4.0, this may have been caused by the changed in 08a8429 as part of #23032. (cc @giuseppe)
Note that moby still includes logic to change the owner of the bundle dir here https://github.com/moby/moby/blob/02a5bd15c001ff0cb45d4366cb9a9deda8293fb8/daemon/internal/libcontainerd/remote/client_linux.go#L53-L86. Although I am aware that Docker does not support the --uidmap my assuption here is that the behaviour in the rootless case (or where the deamon is started with UIDmapping support) is similar to what we would expect from podman.
Steps to reproduce the issue
Run a container with --uidmap and requesting an NVIDIA GPU.
(I could come up with a simple createContainer hook and a dummy CDI spec that would allow this to be reproducible on any machine).
Describe the results you received
The createContainer hook fails to read the container config.
Describe the results you expected
The createContainer hook should read the container config to determine the container root.
podman info output
Reported by internal users and users on the NVIDIA Container Toolkit repo.Podman in a container
No
Privileged Or Rootless
None
Upstream Latest Release
No
Additional environment details
Additional environment details
Additional information
Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting