PVC size not reflecting in guest OS after provisioning

[sagar-86]

What happened:
When provisioning a new VM in KubeVirt, if the volume has a snapshot source and we intend to change the disk size while provisioning, the resulting DV and PVC show new size but OS still reflects the snapshot size. This only happens in a PVC whose volumeMode is set to Block and is not observed in Filesystem mode. The VMs always reflect new size in Filesystem mode. Even restarts, resize2fs and other tools don't recognize the expanded volume. After deployment, if you increase the PVC size by even a single gibibyte, the new size is reflected instantly in OS.

What you expected to happen:
The resultant KubeVirt VM to have the new size as mentioned in the dataVolumeTemplates spec.

How to reproduce it (as minimally and precisely as possible):

• Ensure you have Ubuntu installed block PVC say of size 20G
• Create a snapshot of this block PVC
• Deploy a new VM with dataVolumeTemplates and choose the newly created snapshot as source and use disk size to be 21G
• Observe the DV and PVC of the newly created VM to reflect 21G size
• Log in to the VM and observe the guest OS size in lsblk to reflect 20G

Additional context:
Earlier we were not seeing this issue because we were hitting an issue identified here, which forced a copy instead of snapshot

Environment:

• CDI version (use kubectl get deployments cdi-deployment -o yaml): v1.62.0
• KubeVirt version: v1.5.1
• Kubernetes version (use kubectl version): v1.31.4
• Cloud provider or hardware configuration: Vanilla Kubernetes (bare-metal)
• OS (e.g. from /etc/os-release): Ubuntu 22.04.4 LTS
• Kernel (e.g. uname -a): 5.15.0-94-generic
• Container-runtime: Containerd://1.7.24

[akalenyu]

Can you see the additional space with a regular pod? i.e. blockdev --getsize64 <devicePath>?

[sagar-86]

Still shows 20G

k exec -n <NAMESPACE> <VIRTLAUNCHER_POD> -- blockdev --getsize64 /dev/rootdisk
21474836480

[sagar-86]

I also found out that the PVC resize after provisioning, only reflects in OS if the resize was done with VM powered ON. It doesn't happen if its OFF.

[akalenyu]

k exec -n <VIRTLAUNCHER_POD> -- blockdev --getsize64 /dev/rootdisk
21474836480

If this is the case, it's possible the provisioner does not support online volume resize?

[sagar-86]

k exec -n <VIRTLAUNCHER_POD> -- blockdev --getsize64 /dev/rootdisk
21474836480

If this is the case, it's possible the provisioner does not support online volume resize?

It actually does, that's why I can do resize after the VM is provisioned. The issue is during initial provisioning if target size is != snapshot size. Makes sense?

[akalenyu]

k exec -n <VIRTLAUNCHER_POD> -- blockdev --getsize64 /dev/rootdisk
21474836480

If this is the case, it's possible the provisioner does not support online volume resize?

It actually does, that's why I can do resize after the VM is provisioned. The issue is during initial provisioning if target size is != snapshot size. Makes sense?

Not really tbh, a bit confused. The snippet from the pod shows 20Gi, which means that the expanded storage is not available to the pod

[sagar-86]

Hi @akalenyu, thanks for continuing to dig into this. Let me clarify the scenarios, because I think that’s where the confusion is coming from:

1. Provisioning from snapshot with a larger target size

• Source: Block-mode PVC, 20Gi, snapshot taken.
• Target: New VM created via DataVolumeTemplate with that snapshot as source, but with spec.pvc.resources.requests.storage = 21Gi.
• Observed:
  • The new DV and PVC are indeed created with 21Gi.
  • Inside the virt-launcher pod, blockdev --getsize64 /dev/rootdisk still reports 20Gi.
  • Inside the guest OS (lsblk), it also only shows 20G.
  • Restarting the VM, running resize2fs, etc. does not pick up the extra GiB.

2. Post-deployment resize of the same Block-mode PVC

• If I provision the VM first at 20Gi (matching the snapshot) and then patch the PVC to 21Gi while the VM is running, the resize is recognized immediately:
  • blockdev --getsize64 inside virt-launcher reflects 21Gi.
  • The guest OS can also see the increased capacity.
• If I try the same resize with the VM powered off, it does not propagate into the guest.

---

So the provisioner (Portworx in this case) does support online resize of Block volumes — that’s proven by scenario 2.
The gap is specifically when the initial DV/PVC is created from a snapshot with a larger requested size than the snapshot itself. In that case, the volume object is bigger, but the exposed device path stays at the snapshot’s original size.

---

Question:
Is this a limitation in CDI’s handling of snapshot-sourced DataVolumes (where the resize request is ignored or not passed through properly), or do we need to be handling this differently in our manifests?

Notably, the Filesystem mode case works as expected, so it seems Block mode snapshot restore + resize is where it diverges.

[akalenyu]

Thanks for the detailed explanation!

• Inside the virt-launcher pod, blockdev --getsize64 /dev/rootdisk still reports 20Gi.

This sounds like a bug in the provisioner. The user is asking for 21Gi and the provisioner claims to have satisfied this requirement, however, in reality, it did not (you only have 20Gi in practice).

If I try the same resize with the VM powered off, it does not propagate into the guest.

I see. We'd need to engage someone from Portworx but I don't think it's a limitation on their side tbh.
When you later boot the VM, the storage does not become available (still 20Gi), correct?

Is this a limitation in CDI’s handling of snapshot-sourced DataVolumes (where the resize request is ignored or not passed through properly), or do we need to be handling this differently in our manifests?

I don't think so, but we'll have to see this unravel;
The PVC binding with 21Gi means that 21Gi is really writable, according to k8s spec.
You can reproduce all of these phenomenons without kubevirt in the picture (pod/restored pvc/...)

FYI @pureneelesh @adityadani

[sagar-86]

When you later boot the VM, the storage does not become available (still 20Gi), correct?

Yes

[mhenriks]

Hi @sagar-86 this is really interesting:

If I try the same resize with the VM powered off, it does not propagate into the guest.

You say the provisioner supports "online resize" but I wonder if it supports "offline resize."

Let's try taking KubeVirt out of the picture entirely.

1. Create 20G PVC from snapshot
2. If WFFC, create pod to get the PVC to bind, delete pod
3. Expand the PVC to 21G
4. Create pod that mounts the PVC
5. Check pvc size