Cannot migrate VM back after a host failure

[WanzenBug]

We are testing the integration of our SDS solution with the ManagedBlockStorage/ManagedVolume.

One issue we encountered is the following scenario:

• A VM is running on node A
• Node A fails, becoming unresponsive, we ensure it is shut down
• After oVirt notices the node failure, we restart the VM on node B
• Node A is started again
• Now we try to move the VM back from B to A, but we run into this error:

engine.log:

2025-07-02 16:08:39,374+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand] (default task-22) [4eb12d8a] Command 'AttachManagedBlockStorageVolumeVDSCommand(HostName = node3.olvm.at.linbit.com, AttachManagedBlockStorageVolumeVDSCommandParameters:{hostId='983095aa-c68b-4747-89ae-5725d30c211f', vds='Host[node3.olvm.at.linbit.com,983095aa-c68b-4747-89ae-5725d30c211f]'})' execution failed: VDSGenericException: VDSErrorException: Failed to AttachManagedBlockStorageVolumeVDS, error = Managed Volume is already attached.: "vol_id=5b11e535-7b11-4549-80de-63b1badfc2ad path=/dev/drbd/by-res/volume-5b11e535-7b11-4549-80de-63b1badfc2ad/0 attachment={'type': 'local', 'path': '/dev/drbd/by-res/volume-5b11e535-7b11-4549-80de-63b1badfc2ad/0', 'scsi_wwn': '../../dev/drbd/by-res/volume-5b11e535-7b11-4549-80de-63b1badfc2ad/0'}", code = 927

vdsm.log:

2025-07-02 16:08:39,049+0200 ERROR (jsonrpc/1) [api] FINISH attach_volume error=Managed Volume is already attached.: "vol_id=5b11e535-7b11-4549-80de-63b1badfc2ad path=/dev/drbd/by-res/volume-5b11e535-7b11-4549-80de-63b1badfc2ad/0 attachment=
{'type': 'local', 'path': '/dev/drbd/by-res/volume-5b11e535-7b11-4549-80de-63b1badfc2ad/0', 'scsi_wwn': '../../dev/drbd/by-res/volume-5b11e535-7b11-4549-80de-63b1badfc2ad/0'}" (api:114)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolumedb.py", line 170, in add_volume
    self._conn.execute(sql, (vol_id, conn_info_json))
sqlite3.IntegrityError: UNIQUE constraint failed: volumes.vol_id

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 212, in _add_volume
    db.add_volume(vol_id, connection_info)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolumedb.py", line 172, in add_volume
    raise VolumeAlreadyExists(vol_id, connection_info)
vdsm.storage.managedvolumedb.VolumeAlreadyExists: Failed to store {'driver_volume_type': 'local', 'data': {'device_path': '/dev/drbd/by-res/volume-5b11e535-7b11-4549-80de-63b1badfc2ad/0', 'access_mode': 'rw'}}.Volume with id 5b11e535-7b11-4549-80de-63b1badfc2ad already exists in the DB

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 107, in method
    ret = func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/API.py", line 1803, in attach_volume
    connection_info)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 66, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 91, in attach_volume
    _add_volume(db, vol_id, connection_info)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 221, in _add_volume
    vol_id, vol_info["path"], vol_info.get('attachment'))
vdsm.storage.exception.ManagedVolumeAlreadyAttached: Managed Volume is already attached.: "vol_id=5b11e535-7b11-4549-80de-63b1badfc2ad path=/dev/drbd/by-res/volume-5b11e535-7b11-4549-80de-63b1badfc2ad/0 attachment={'type': 'local', 'path': '/dev/drbd/by-res/volume-5b11e535-7b11-4549-80de-63b1badfc2ad/0', 'scsi_wwn': '../../dev/drbd/by-res/volume-5b11e535-7b11-4549-80de-63b1badfc2ad/0'}"
2025-07-02 16:08:39,056+0200 INFO  (jsonrpc/1) [api.storage] FINISH attach_volume return={'status': {'code': 927, 'message': 'Managed Volume is already attached.: "vol_id=5b11e535-7b11-4549-80de-63b1badfc2ad path=/dev/drbd/by-res/volume-5b11e535-7b11-4549-80de-63b1badfc2ad/0 attachment={\'type\': \'local\', \'path\': \'/dev/drbd/by-res/volume-5b11e535-7b11-4549-80de-63b1badfc2ad/0\', \'scsi_wwn\': \'../../dev/drbd/by-res/volume-5b11e535-7b11-4549-80de-63b1badfc2ad/0\'}"'}} from=::ffff:10.43.235.10,49886, flow_id=4eb12d8a (api:37)
2025-07-02 16:08:39,056+0200 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call ManagedVolume.attach_volume failed (error 927) in 0.01 seconds (__init__:301)

Now it looks like vdsm still has the "attached" volumes from before the reboot present, which causes the failure here.

After manually removing the attachment using vdsm-client ManagedVolume detach_volume, the migration works without issues.

I guess one fix would be to compare the new attachment to the existing one, and ignore if they are the same. Another option would be to wipe the database on reboot (i.e. putting it in /run). I can't think of a case where the database needs to be kept after a reboot, but I am also not an expert on vdsm. 😉