[BUG] MountVolume.MountDevice failed for volume

[Awadabang]

What happened:

pod启动报错：Warning FailedMount 4m43s (x41 over 85m) kubelet MountVolume.MountDevice failed for volume "pvc-6aaff28b-43e5-431a-b5aa-1a79c7846fa2" : rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: read unix @->/csi/csi.sock: use of closed network connection"

What you expected to happen:

正常启动pod

How to reproduce it (as minimally and precisely as possible):

就这个节点会这样，而且很多僵尸进程，都是jfs-mount相关的

Anything else we need to know?
kubelet日志：

Dec 29 10:55:03 yai-prod-k8s-gpu-node-05 kubelet[4417]: W1229 10:55:03.683705    4417 logging.go:59] [core] [Channel #16784629 SubChannel #16784630] grpc: addrConn.createTransport failed to connect to {Addr: "/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock", ServerName: "localhost", }. Err: connection error: desc = "error reading server preface: read unix @->/csi/csi.sock: use of closed network connection"
Dec 29 10:55:03 yai-prod-k8s-gpu-node-05 kubelet[4417]: I1229 10:55:03.683861    4417 trace.go:236] Trace[524000438]: "Calculate volume metrics of vol-workspace-d20ti12011bs73d318o0 for pod prod-yai/env-d20ti12011bs73d318ng" (29-Dec-2025 10:54:43.682) (total time: 20000ms):
Dec 29 10:55:03 yai-prod-k8s-gpu-node-05 kubelet[4417]: Trace[524000438]: [20.000915724s] [20.000915724s] END
Dec 29 10:55:05 yai-prod-k8s-gpu-node-05 kubelet[4417]: W1229 10:55:05.944208    4417 logging.go:59] [core] [Channel #16784632 SubChannel #16784633] grpc: addrConn.createTransport failed to connect to {Addr: "/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock", ServerName: "localhost", }. Err: connection error: desc = "error reading server preface: read unix @->/csi/csi.sock: use of closed network connection"
Dec 29 10:55:05 yai-prod-k8s-gpu-node-05 kubelet[4417]: I1229 10:55:05.944384    4417 trace.go:236] Trace[829666322]: "Calculate volume metrics of vol-workspace-d2s287a011bs73dqugi0 for pod prod-yai/env-d2s287a011bs73dqughg" (29-Dec-2025 10:54:45.943) (total time: 20001ms):
Dec 29 10:55:05 yai-prod-k8s-gpu-node-05 kubelet[4417]: Trace[829666322]: [20.001080238s] [20.001080238s] END
Dec 29 10:55:09 yai-prod-k8s-gpu-node-05 kubelet[4417]: W1229 10:55:09.926584    4417 logging.go:59] [core] [Channel #16784635 SubChannel #16784636] grpc: addrConn.createTransport failed to connect to {Addr: "/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock", ServerName: "localhost", }. Err: connection error: desc = "error reading server preface: read unix @->/csi/csi.sock: use of closed network connection"
Dec 29 10:55:09 yai-prod-k8s-gpu-node-05 kubelet[4417]: I1229 10:55:09.926724    4417 trace.go:236] Trace[1662259402]: "Calculate volume metrics of dataset for pod prod-yai/env-d4ska2q011bs73brhu10" (29-Dec-2025 10:54:49.926) (total time: 20000ms):
Dec 29 10:55:09 yai-prod-k8s-gpu-node-05 kubelet[4417]: Trace[1662259402]: [20.000514548s] [20.000514548s] END

Environment:

• JuiceFS CSI Driver version (which image tag did your CSI Driver use): v0.25.0
• Kubernetes version (e.g. kubectl version): v1.30
• Object storage (cloud provider and region): minio
• Metadata engine info (version, cloud provider managed or self maintained): tikv
• Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage):
• Others:

[zwwhdls]

对应节点的 csi node 是不是退出了

[Awadabang]

对应节点的 csi node 是不是退出了

没有退出，但是有报错：
juicefs-plugin容器：

E1229 01:04:10.349361       7 reconciler.go:150] "reconciler-controller: Driver check pod error, will retry" err="exec cmd: mount -o bind /jfs/pvc-4be3354c-0816-456c-b408-5ca6f3429b4c-hpyxzv/pvc-4be3354c-0816-456c-b408-5ca6f3429b4c /var/lib/kubelet/pods/99c548c4-cefd-4364-bbcc-ccae197c14e8/volumes/kubernetes.io~csi/pvc-4be3354c-0816-456c-b408-5ca6f3429b4c/mount err:function timeout" name="juicefs-yai-prod-k8s-gpu-node-05-pvc-4be3354c-0816-456c-b408-5ca6f3429b4c-sizjnt"
I1229 01:04:22.072110       7 node.go:306] "NodeGetVolumeStats: Check volume path %s, err: %s" volumePath="/var/lib/kubelet/pods/7665ee12-e3dc-4fa9-92fa-80029c9a2572/volumes/kubernetes.io~csi/pvc-3e2bebee-dfa5-47f9-9354-71b76c7c3a0e/mount" error="function timeout"
E1229 01:04:22.073637       7 driver.go:105] "driver: GRPC error" err="rpc error: code = Internal desc = Check volume path, err: function timeout"
I1229 01:04:42.123076       7 pod.go:242] "pod-driver: waiting for mount point ready" podName="juicefs-yai-prod-k8s-gpu-node-05-pvc-3e2bebee-dfa5-47f9-9354-71b76c7c3a0e-ccbsbk"
I1229 01:04:42.123944       7 pod.go:261] "pod-driver: Mount point is ready" podName="juicefs-yai-prod-k8s-gpu-node-05-pvc-3e2bebee-dfa5-47f9-9354-71b76c7c3a0e-ccbsbk"
I1229 01:04:43.124635       7 pod_driver.go:669] "pod-driver/recover: umount pod target before recover" podName="juicefs-yai-prod-k8s-gpu-node-05-pvc-3e2bebee-dfa5-47f9-9354-71b76c7c3a0e-ccbsbk" target="/var/lib/kubelet/pods/7665ee12-e3dc-4fa9-92fa-80029c9a2572/volumes/kubernetes.io~csi/pvc-3e2bebee-dfa5-47f9-9354-71b76c7c3a0e/mount" remain mount count=5
I1229 01:04:44.128381       7 pod_driver.go:687] "pod-driver/recover: recover volPath" podName="juicefs-yai-prod-k8s-gpu-node-05-pvc-3e2bebee-dfa5-47f9-9354-71b76c7c3a0e-ccbsbk" target="/var/lib/kubelet/pods/7665ee12-e3dc-4fa9-92fa-80029c9a2572/volumes/kubernetes.io~csi/pvc-3e2bebee-dfa5-47f9-9354-71b76c7c3a0e/mount" mountPath="/jfs/pvc-3e2bebee-dfa5-47f9-9354-71b76c7c3a0e-tuawvk/pvc-3e2bebee-dfa5-47f9-9354-71b76c7c3a0e"
E1229 01:04:45.128952       7 pod_driver.go:694] "pod-driver/recover: bind mount error" err="function timeout" podName="juicefs-yai-prod-k8s-gpu-node-05-pvc-3e2bebee-dfa5-47f9-9354-71b76c7c3a0e-ccbsbk"
E1229 01:04:45.128986       7 reconciler.go:150] "reconciler-controller: Driver check pod error, will retry" err="exec cmd: mount -o bind /jfs/pvc-3e2bebee-dfa5-47f9-9354-71b76c7c3a0e-tuawvk/pvc-3e2bebee-dfa5-47f9-9354-71b76c7c3a0e /var/lib/kubelet/pods/7665ee12-e3dc-4fa9-92fa-80029c9a2572/volumes/kubernetes.io~csi/pvc-3e2bebee-dfa5-47f9-9354-71b76c7c3a0e/mount err:function timeout" name="juicefs-yai-prod-k8s-gpu-node-05-pvc-3e2bebee-dfa5-47f9-9354-71b76c7c3a0e-ccbsbk"
I1229 01:05:21.344991       7 node.go:306] "NodeGetVolumeStats: Check volume path %s, err: %s" volumePath="/var/lib/kubelet/pods/99c548c4-cefd-4364-bbcc-ccae197c14e8/volumes/kubernetes.io~csi/pvc-4be3354c-0816-456c-b408-5ca6f3429b4c/mount" error="function timeout"
E1229 01:05:21.404680       7 driver.go:105] "driver: GRPC error" err="rpc error: code = Internal desc = Check volume path, err: function timeout"

liveness-probe 容器：

I0901 07:13:35.898250       1 main.go:133] "Calling CSI driver to discover driver name"
I0901 07:13:35.901601       1 main.go:141] "CSI driver name" driver="csi.juicefs.com"
I0901 07:13:35.901623       1 main.go:170] "ServeMux listening" address="0.0.0.0:9909"
E1207 03:58:04.414703       1 main.go:67] "Failed to establish connection to CSI driver" err="context deadline exceeded"
E1207 04:08:14.414662       1 main.go:67] "Failed to establish connection to CSI driver" err="context deadline exceeded"
E1207 04:08:24.414175       1 main.go:67] "Failed to establish connection to CSI driver" err="context deadline exceeded"
E1207 04:08:34.414552       1 main.go:67] "Failed to establish connection to CSI driver" err="context deadline exceeded"
E1207 04:08:44.414755       1 main.go:67] "Failed to establish connection to CSI driver" err="context deadline exceeded"
E1207 04:08:54.413711       1 main.go:67] "Failed to establish connection to CSI driver" err="context deadline exceeded"
E1229 01:05:14.414860       1 main.go:77] "Health check failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1229 01:05:34.413679       1 main.go:77] "Health check failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1229 01:05:44.417885       1 main.go:77] "Health check failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1229 01:05:54.414972       1 main.go:67] "Failed to establish connection to CSI driver" err="context deadline exceeded"
E1229 01:06:04.414718       1 main.go:67] "Failed to establish connection to CSI driver" err="context deadline exceeded"
E1229 01:06:14.414983       1 main.go:67] "Failed to establish connection to CSI driver" err="context deadline exceeded"

node-driver-registrar容器：

I1229 03:04:29.132124       1 connection.go:249] GRPC response: {"endpoint":"/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock","name":"csi.juicefs.com","supported_versions":["1.0.0"],"type":"CSIPlugin"}
I1229 03:04:29.132133       1 connection.go:250] GRPC error: <nil>
I1229 03:04:29.132185       1 node_register.go:106] health check succeeded
I1229 03:04:39.131378       1 node_register.go:133] Attempting to open a gRPC connection with: "/registration/csi.juicefs.com-reg.sock"
I1229 03:04:39.131414       1 connection.go:213] Connecting to unix:///registration/csi.juicefs.com-reg.sock
I1229 03:04:39.131760       1 node_register.go:141] Calling node registrar to check if it still responds
I1229 03:04:39.131774       1 connection.go:242] GRPC call: /pluginregistration.Registration/GetInfo
I1229 03:04:39.131777       1 connection.go:243] GRPC request: {}
I1229 03:04:39.131924       1 main.go:90] Received GetInfo call: &InfoRequest{}
I1229 03:04:39.132075       1 connection.go:249] GRPC response: {"endpoint":"/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock","name":"csi.juicefs.com","supported_versions":["1.0.0"],"type":"CSIPlugin"}
I1229 03:04:39.132079       1 connection.go:250] GRPC error: <nil>
I1229 03:04:39.132139       1 node_register.go:106] health check succeeded
I1229 03:04:49.131299       1 node_register.go:133] Attempting to open a gRPC connection with: "/registration/csi.juicefs.com-reg.sock"
I1229 03:04:49.131347       1 connection.go:213] Connecting to unix:///registration/csi.juicefs.com-reg.sock
I1229 03:04:49.131824       1 node_register.go:141] Calling node registrar to check if it still responds
I1229 03:04:49.131837       1 connection.go:242] GRPC call: /pluginregistration.Registration/GetInfo
I1229 03:04:49.131841       1 connection.go:243] GRPC request: {}
I1229 03:04:49.132028       1 main.go:90] Received GetInfo call: &InfoRequest{}
I1229 03:04:49.132183       1 connection.go:249] GRPC response: {"endpoint":"/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock","name":"csi.juicefs.com","supported_versions":["1.0.0"],"type":"CSIPlugin"}
I1229 03:04:49.132192       1 connection.go:250] GRPC error: <nil>
I1229 03:04:49.132254       1 node_register.go:106] health check succeeded```