[BUG] Volumes are not (always) expanding

[okdas]

Describe the bug

For some reason, some volumes are not getting expanded.

To Reproduce

• Bump storage size on PVC that is attached to an active pod
• df -h shows old size
• Restart the pod to make sure lack of restart is the reason for this issue
• df -h shows old size

Expected behavior

Volumes to expand when they're marked as so via k8s events.

Additional context
Here's an example, I tried doing so twice with this volume:

Name:          testnet-beta-fullnode2-pocketd-data
Namespace:     testnet-beta
StorageClass:  vultr-block-storage
Status:        Bound
Volume:        pvc-32b95169a43744cc
Labels:        <none>
Annotations:   argocd.argoproj.io/tracking-id: testnet-beta-fullnode2:/PersistentVolumeClaim:testnet-beta/testnet-beta-fullnode2-pocketd-data
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: block.csi.vultr.com
               volume.kubernetes.io/storage-provisioner: block.csi.vultr.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      301Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       testnet-beta-fullnode2-pocketd-0
Events:
  Type    Reason                      Age                From                                  Message
  ----    ------                      ----               ----                                  -------
  Normal  ExternalExpanding           10m (x2 over 23h)  volume_expand                         waiting for an external controller to expand this PVC
  Normal  Resizing                    10m                external-resizer block.csi.vultr.com  External resizer is resizing volume pvc-32b95169a43744cc
  Normal  FileSystemResizeRequired    10m                external-resizer block.csi.vultr.com  Require file system resize of volume on node
  Normal  FileSystemResizeSuccessful  10m (x2 over 23h)  kubelet                               MountVolume.NodeExpandVolume succeeded for volume "pvc-32b95169a43744cc" protocol-k8s-nj-4c-32g-fc844061879a

Though it is marked as expand succeded, the filesystem didn't expand:

pocket@testnet-beta-fullnode2-pocketd-0:/$ df -h | grep virtio
/dev/disk/by-id/virtio-ewr-9e7c1c61938c4d  246G  208G   38G  85% /home/pocket/.pocket

I reproduced this behavior on both v0.14.0 csi, and on v0.16.0.

[okdas]

Found a way to force expansion: by evicting the pod off the node. The size of the filesystem changes when it gets assigned to another node.

Is there a way to make filesystem grow without eviction? Thanks.

[optik-aper]

This is a limitation with the virtualization layer on Vultr VMs which we're looking into. In the meantime, I'm going to rework the node expansion logic to hopefully make a more informative expansion process for block storages.