hf_transfer download sometimes hangs without failing or succeeding

[chamini2]

System Info

hf_transfer = 0.1.6
huggingface_hub = 0.21.4

Reproduction

We have not been able to reproduce and it is hard to really give details, since it happens very rarely in a system that deletes the downloaded file after using it, so no real way to know the state it leaves the machine in. But the behaviour is that we will start a download and it just will never return, the callback is not called either, nothing.

Eventually our system closes the process and we do not have visibility into it other than the logs it produces. The lib produces none logs, only some logs from our code:

Downloading https://xxx.r2.cloudflarestorage.com/yyy to /tmp/file

and there it hangs

Expected behavior

It should either fail or succeed, but not hang

---

Thanks for this amazing project, by the way. The speed is just amazing! I wish I was able to give more hints, but it really shows us nothing.

[aikitoria]

This sometimes happens for me as well. Maybe 1% of downloads. Makes huggingface-cli freeze up, have to kill the process to retry.

[chamini2]

I moved away from using this tool and some times I have slow downloads from hugging face. So maybe it is HF and not the tool.

[aikitoria]

Happened again. Froze with a file at 99% and locked up the whole session. Had to switch to another and kill the process to retry, which immediately worked.

[McPatate]

Could be a lot of different things, will need more information to identify what the issue is. Is your network connectivity stable? Having a reproducible example would help a lot.

[aikitoria]

As far as I can tell, there is no way to reproduce it on purpose. Even trying the same file again right after it happens usually works totally fine.

[chamini2]

I actually think it may be huggingface rate limiting, since I saw it happen for other tools too.

[chamini2]

I just set a timeout for the download and retry

[McPatate]

@chamini2 that would definitely explain the issue.

[aikitoria]

Would it? That seems like a bug. If HF has a rate limit, then an official tool from HF should either be aware of the rate limit and automatically stay under it, or detect the rate limit response and automatically retry.

Also, it wouldn't explain why the download stops at exactly 99%.

[McPatate]

@aikitoria cf the Disclaimer. hf_transfer was meant to be used internally at first, where rate limit is not an issue.

If you want to reduce the number of requests, I would play with the following two settings:

    chunk_size=CHUNK_SIZE,
    max_files=64,

max_files is the number of concurrent chunks that are downloaded at once, and chunk_size is the size in bytes of a chunk. Ofc, 10Mib seems like a sweet spot, but increasing this number will reduce the total number of calls required to download the file and thus reduce chances of getting rate limited. Similarly, reducing the number of concurrent chunk downloads will delay hitting the rate limit.

Download speed will probably suffer a bit, but I don't think it'll be too bad.

We could detect we are getting a 429 response code and throttle ourself, but I'm not sure this really is in scope for this project. cc @Narsil wdyt?

[McPatate]

Also, it wouldn't explain why the download stops at exactly 99%.

It'd make sense that download would stop at higher percentages given a lot of requests are emitted at once, depending on the file size though. As previously mentioned, we would need more information to debug this issue correctly.

[aikitoria]

How can we collect this information, given the issue seems to happen randomly with a low chance? I still found no trick to it. It actually hasn't happened in a bit.

Are there some verbose debug logs we can enable all the time - and then grab if it happens?

[McPatate]

Sadly we don't have any tracing related to hf_transfer, you will have to add it yourself. Are you using the python bindings or rust functions directly?

If you add tracing to hf_transfer, you will also need to create a Subscriber using tracing-subscriber.

This way, you can set RUST_LOG=trace <command> <to> <run> <my> <download> for instance and you will see a lot of info about what is going on.

[Narsil]

We could detect we are getting a 429 response code and throttle ourself, but I'm not sure this really is in scope for this project. cc @Narsil wdyt?

Definitely feels out of scope to me. (also what do you do, and how much do you throttle yourself ?, I'm not sure there are good answers here)

[aikitoria]

I don't see how it could be out of scope. hf_transfer is the code that receives the rate limit response for a chunk, so the only reasonable option is to implement it there (you don't want to re-download the entire file!).

Usually 429 includes metadata on when to retry. If not, infinite retries (until user cancel) with exponential backoff always works.