Pod Stuck in CreateContainerConfigError When Sharing a Volume on the Same Host

[wolgod]

What happened:

I'm encountering an issue where multiple pods from different deployments are scheduled on the same node and use a shared volume. Each pod mounts a different subdirectory of this shared volume.

When one of these pods unexpectedly restarts, it gets stuck in a CreateContainerConfigError state. Upon inspection, it appears that the pods are using the same mount point on the host, but for different subdirectories.

The root cause seems to be a conflict where the mount point for the restarting pod is still being held by another pod, preventing it from being unmounted, remounted, and started. The lsof command on the host shows the mount point is in use by another pod.

The specific error log is as follows:
kubelet: Unmounting arguments: /var/lib/kubelet/pods/a5511304-bcf1-4a3e-abe6-c6bc22aca1e4/volume-subpaths/fs-545358898194812475-106296-pv/sub-volume/0
kubelet: Output: umount: /var/lib/kubelet/pods/a5511304-bcf1-4a3e-abe6-c6bc22aca1e4/volume-subpaths/fs-545358898194812475-106296-pv/sub-volume/0: target is busy.
kubelet: (In some cases useful info about processes that use
kubelet: the device is found by lsof(8) or fuser(1))

pod_workers.go:191] Error syncing pod a5511304-bcf1-4a3e-abe6-c6bc22aca1e4 ("sub-remove-subremove-test-b78cdcb6d-tknmq_ns-106296(a5511304-bcf1-4a3e-abe6-c6bc22aca1e4)"), skipping: failed to "StartContainer" for "sub-volume" with CreateContainerConfigError: "failed to prepare subPath for volumeMount "fs-545358898194812475" of container "sub-volume""

What you expected to happen:
pod restart nornaml

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?

Environment:

• JuiceFS CSI Driver version (which image tag did your CSI Driver use): 0.25.4
• Kubernetes version (e.g. kubectl version): 1.20
• Object storage (cloud provider and region): ceph rbd
• Metadata engine info (version, cloud provider managed or self maintained):
• Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage):
• Others:

[wolgod]

When I destroy a pod, it also gets stuck in the Terminating state. The pod can only be terminated successfully if all other pods on the node using the same shared PVC are also deleted.

[zxh326]

unable to reproduce in my environment, could you describe the reproduction steps in detail? It would be best if you could also attach a YAML file.

[wolgod]

mount like this,different deployments mount different subpaths. When pods from multiple different deployments are scheduled to the same host, some pods may output real-time log files to subdirectories under their respective subpaths.

      volumeMounts:
      - mountPath: /models/Qwen3-32B-1
        mountPropagation: HostToContainer
        name: fs-545358898194812475
        subPath: nlp/models/Qwen3-32B-1
 ........
    volumes:
    - name: fs-545358898194812475
      persistentVolumeClaim:
        claimName: fs-545358898194812475

[zxh326]

Still unable to reproduce... Each pod has its own mount directory, even if they use the same PVC; you can see in the logs that each path includes the corresponding pod uid.

kubelet: Unmounting arguments: /var/lib/kubelet/pods/a5511304-bcf1-4a3e-abe6-c6bc22aca1e4/volume-subpaths/fs-545358898194812475-106296-pv/sub-volume/0

[wolgod]

Still unable to reproduce... Each pod has its own mount directory, even if they use the same PVC; you can see in the logs that each path includes the corresponding pod uid.

kubelet: Unmounting arguments: /var/lib/kubelet/pods/a5511304-bcf1-4a3e-abe6-c6bc22aca1e4/volume-subpaths/fs-545358898194812475-106296-pv/sub-volume/0

I have a single juicefs mount pod that handles the mounting of a volume on the host. All of the other pods that need to access this volume do so by bind-mounting the same single host mount point, even though they might be running with different user IDs (UIDs). Is that correct?

when lsof /var/lib/kubelet/pods/a5511304-bcf1-4a3e-abe6-c6bc22aca1e4/volume-subpaths/fs-545358898194812475-106296-pv/sub-volume/0

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
pt_main_t 43180 root 19w REG 0,170 5714 336344154 /nlp_models/Qwen2.5-72B-Int4/logs/qwen_1752572259.346269-2025-08-31.log
pt_main_t 43181 root 19w REG 0,170 18652 336344155 /nlp_models/Qwen2.5-72B-Int4/logs/qwen_1752572259.5348663-2025-08-31.log
pt_main_t 43182 root 23w REG 0,170 2174 336344157 /nlp_models/Qwen2.5-72B-Int4/logs/qwen_1752572259.410958-2025-08-31.log
pt_main_t 43183 root 18w REG 0,170 8741 336344156 /nlp_models/Qwen2.5-72B-Int4/logs/qwen_1752572259.433411-2025-08-31.log

/nlp_models is a directory mounted into the container from another pod.