Is there any way to enable multiple threads when adding files?

[FrostyLeaf]

We found DVC could be insanely slow when adding lots of small files (100k files, each one has roughly 150KiB size) since it calc hash and transfer files with only one thread.

$ time dvc add images/
Adding...
Bus error (core dumped)erring                                 67.9k/118k [18:59<14:39,  57.2file/s]

real    41m10.129s
user	37m54.970s
sys	1m49.204s

As you can see, it took more than 40 minutes to calc hash and transfer files, and eventually... failed.

We also tested this dataset with Git LFS, it only took 5m10s to finish adding all of the image files. As Git itself, Git LFS always tends to use multiple threads to handle heavy jobs.

[daavoo]

You can set the number of threads for hashing with the option core.checksum_jobs of dvc config . See https://dvc.org/doc/command-reference/config#core

[FrostyLeaf]

I have 4 CPU cores. According to the docs, it should be 2 by default, so there is already more than 1 thread? However, when I checked In top, DVC always took ~ 100% CPU.

[shcheklein]

@FrostyLeaf could you please run dvc version and share its output please? to give us a bit more information. Have you tried --job or core.checksum_jobs as @daavoo suggested?

Also would be great if you could run https://github.com/iterative/dvc/wiki/Debugging,-Profiling-and-Benchmarking-DVC on a smaller subset of files to see why is it happening.

[daavoo]

I can see that it's utilizing singe thread

I tested locally and it is using 4 threads by default on my machine (from max(1, min(4, cpu_count() // 2))).

I traced with viztracer:

setup.sh

rm -rf .dvc
rm -rf foo.dvc
rm -rf foo
dvc init
mkdir foo
for i in {0..10}
do
    echo i > "foo/$(printf "%03d" "$i").txt"
done

test_add.py

from dvc.api import Repo

repo = Repo(".")
repo.add("foo")

Run: bash setup.sh && viztracer test_add.py --max_stack_depth 10 && vizviewer result.json

[daavoo]

Not sure how useful this is but, from the profile, what I can tell is that it appears that state.get and state.save used inside get_file_hash are blocking the threads. Hardcoding the number of threads to 1 gives the same or better performance.

Following the stack, it seems to be due to calls of internal sqlite3 used by our dependency diskcache for storing the md5s in State .

[shcheklein]

Good stuff, @daavoo !

Can be related - #7129 ?

When did we add the diskcache and is there a way to get the current DB schema (including primary keys and other indexes) + is there a way to enable debug logs to see actual queries it's running and their time.

Can we get the DVC version before and after diskcache and compare them?

[shcheklein]

(before == the one that explicit indexes and other optimizations. It might be the thing that is very common to ORM systems - unless we know what we do they are far from being optimal to use)

[efiop]

The state situation is known, we are not using it efficiently. diskcache was added in 2.0. Guys, please don't waste your time on this, we are aware and will be getting to it as planned.