oci-bv - Timed out waiting for backup to become available

[martysweet]

Hi,

We are using a VolumeSnapshotClass as below for Block volume snapshotting:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: oci-bv-snapshot-incremental
driver: blockvolume.csi.oraclecloud.com
parameters:
  backupType: incremental # No functional restore difference between full and incremental
deletionPolicy: Delete

This is integrated with CNPG for lower environment database volume snapshots.
Occasionally (every few weeks), we find these backups failing. with the error:

DeadlineExceeded desc = Timed out waiting for backup to become available

It looks like this is being thrown by the oci-bv csi here:

oci-cloud-controller-manager/pkg/csi/driver/bv_controller.go

Line 1099 in 411bfeb

return nil, status.Errorf(codes.DeadlineExceeded, "Timed out waiting for backup to become available")

Which uses a timeout of 45 seconds as defined here:

oci-cloud-controller-manager/pkg/csi/driver/bv_controller.go

Line 1068 in 411bfeb

volumeAvailableTimeoutCtx, cancel := context.WithTimeout(ctx, 45 * time.Second)

However, in practice a 45 second timeout is too conservative, looking in the logs, we see the following times for snapshot creation in uk-london-1 between going from com.oraclecloud.BlockVolumes.CreateVolumeBackup.begin to com.oraclecloud.BlockVolumes.CreateVolumeBackup.end state.

Over 9 samples: average: 37.4 seconds | min: 34 seconds | max: 41 seconds

With a backupPollInterval of 5 seconds, the CSI steps just outside of the permissible timeout of 45 seconds.
https://github.com/oracle/oci-cloud-controller-manager/blob/master/pkg/oci/client/block_storage.go#L150C36-L150C60

https://github.com/oracle/oci-cloud-controller-manager/blob/master/pkg/oci/client/block_storage.go#L42

I believe the solution for this would be to increase the available timeout to 60 seconds to align better with the expected response times from the API.

oci-cloud-controller-manager/pkg/csi/driver/bv_controller.go

Line 1068 in 411bfeb

volumeAvailableTimeoutCtx, cancel := context.WithTimeout(ctx, 45 * time.Second)

Thanks!

[ms3rgio]

We are facing the same problem in the same situation.

[silvio89]

Same problem here, also using CNPG. However, I believe that 60 seconds won't be enough for us. We have an 8TB and a 20TB disks that takes longer than that to become available. None of our attempts with snapshots on this database cluster have been successful. The smaller ones are working fine.