What are the files that haystack downloads to cache directory? and how can I create a local package?

[asharm0662]

Hi,

When I am using a new model in the reader variable like so:

reader = TransformersReader(tokenizer="deepset/roberta-base-squad2", use_gpu=-1)

I notice there are files downloaded to ~/.cache/hugging/transformers/.

1. are these files static?
2. what are these files exactly?

I tried downloading the model locally from hugging face using these steps:

1. https://huggingface.co/distilbert-base-uncased
2. git lfs install
3. git clone https://huggingface.co/distilbert-base-uncased

But I still need what is in the /.cache directory. How can I create a local hugging face model/ cache files for deployment in cloud?

[julian-risch]

Hi @asharm0662
the files downloaded to ~/.cache/hugging/transformers/ are language models that interpret queries and find an answer in a text document (for example) and they include the corresponding tokenizers that split any arbitrary input strings, e.g., queries, into sequences of tokens. When you run reader = TransformersReader(tokenizer="deepset/roberta-base-squad2", use_gpu=-1) the tokenizer is loaded from https://huggingface.co/deepset/roberta-base-squad2
These files are about 2 GB large, sometimes even larger. We cache these models inside the transformers library because it would take a long time to download them on-the-fly every time you want to run a query. The models won't change so they are static, yes.

If you would like to fill the cache, I recommend that you have a look at how we create our docker files with the models already cached: https://github.com/deepset-ai/haystack/pull/1978/files
Another way is to copy the cache from a different machine or to save the model on disk: https://stackoverflow.com/questions/62261602/downloading-transformers-models-to-use-offline

However, when you run your cloud-deployed application for the first time (with internet connection) the cache will be filled automatically. Is there any particular reason why you want to make sure that the model is already in the cache even before using it for the first time?

[asharm0662]

this worked thank you.

[OlivierBondu]

Hi !

I have a similar use-case and to give a bit more context into why I need to do this: I need to wrap up the python code into a docker images for portability. However the docker container will run on an infrastructure that cannot access the internet. Thus, I need to embed the models in the docker image at build time, using the caching mechanism above.

I hope some day we would be able to run some kind of huggingface mirror (à la pypi) so that we can keep the docker images to their minimum and download the models only when needed and from a trusted internal repo, but in the meantime this is the easiest workaround.

However, I cannot have this workaround work at the moment and my application dies after the timeout to huggingface... any idea ?

Cheers,

[anakin87]

Hello @OlivierBondu!

In the past, we prepared a repository to cover similar uses cases: Airgapped REST API.

Have a look and see if it helps you...
If you need more help, please open another discussion (we don't usually monitor older ones).