DVC corrupts database file silently and causes loss of data

[alita-moore]

I just pushed a database file using dvc using the standard dvc add file.db then dvc push file.db etc. and there were no errors, except when you pull the database from the remote the resolved database is corrupted. This is obviously very bad and could have cost our team thousands of dollars or more. I'd like to identify how this happened and figure out how to never let it happen again.

The machine that pushed the original database has the correct database in its cache, dvc push claims that all files were pushed, but when downloading the database onto another machine the database is malformed. I'd be happy to hop on a call or something as well to help debug this / find the root cause in our dataset.

Thankfully this did not result in us losing data, but this could very easily not be the case in the future. Thank you for your help with this.

here is my output from dvc doctor which may be helpful:

dvc doctor
DVC version: 3.55.2 (pip)
-------------------------
Platform: Python 3.10.13 on Linux-6.8.0-1029-aws-x86_64-with-glibc2.36
Subprojects:
        dvc_data = 3.16.6
        dvc_objects = 5.1.0
        dvc_render = 1.0.2
        dvc_task = 0.40.2
        scmrepo = 3.3.8
Supports:
        http (aiohttp = 3.10.9, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.10.9, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2024.9.0, boto3 = 1.35.36)
Config:
        Global: /home/woven/.config/dvc
        System: /etc/xdg/dvc
Cache types: symlink
Cache directory: ext4 on /dev/root
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/root
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/4124a8e904743eedc398651576f18218

You are using dvc version 3.55.2; however, version 3.60.0 is available.
To upgrade, run 'pip install --upgrade dvc'.

[skshetry]

Maybe related to #10767?

Please try updating s3fs to the latest version.

[alita-moore]

I think it's just a coincidence (good timing, though), but I can't say anything with certainty. I only checked the version today. I have also been experiencing this problem a few times in the past as well but assumed I just did something wrong.

[alita-moore]

from what I can tell the error I was seeing earlier was primarily a result of having set the "verify" configuration option in my s3 remote. From what I can tell though this only validates when pulling from the remote. In other words, if machine A commits an object and then pushes it, then they have the object in their local cache, because they have it in their local cache there is no need to download from remote. Given that the object they pushed was actually corrupted by the remote in some way, it would lead to machine A thinking that the remote has a valid object store. Then machine B would download from the file store, it would be verified, and the system would detect that something is corrupted. However, at this point machine A may have already deleted their local cache (which can be possible if machine A were an EC2 instance that was terminated). Therefore, it's possible to silently lose data.

This verify task should be run at the time of pushing the file as well. It needs to re-fetch the file from the remote and verify the hash. Or perhaps you can think of a better way.

Either way, I think in its current form it is possible to lose data without warning, which in my opinion is a fundamental flaw with dvc's core purpose and it may be helpful to patch that sooner rather than later.

This also means that whatever caused the remote corruption may have not been the issue you mentioned. So I suspect it would also be necessary in the case that the remote is corrupt that it allow for retrying / force-pushing an updated file to the remote. This seems critical for the safe operation of the dvc tool, given that its primary function is managing the remote and local store of data.

[alita-moore]

pushing the new version of the database with a small change worked. so my issue is resolved, thank you for your help on this.

[skshetry]

@alita-moore, could you check the file size (in bytes) in both the cache and the remote? If you share those, I can tell whether it's related to that bug.

[alita-moore]

the file size was different between the two, it's gone now though so I can't confirm but I did check the byte size in the version from the remote and the version locally and they were different.