EmbeddingRetriever with custom trained models

[sankalp-acl]

How can I use EmbeddingRetriever with a model that I have trained? I'm using the below code to initialize the retriever.

retriever = EmbeddingRetriever(
        document_store=document_store,
        embedding_model=model, # this is my custom-trained model
        model_format="sentence_transformers",
    )

I am loading my custom-trained model and tokenizer as follows. The custom model is a pretrained hugging face sentence-transformer model that is fine-tuned on some dataset.

model = AutoModelForSequenceClassification.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

[anakin87]

Hello @sankalp-acl your approach seems good. Have you encountered any issues?

[sankalp-acl]

@anakin87 Yes. I'm getting the below error:

FileNotFoundError: No such file or directory: '/tmp/model/processor_config.json'

[anakin87]

Related to #1801.

[sankalp-acl]

@anakin87 I saw the above-linked issue (#1801). However, I did not see any resolution for it. Please let me know if there's a fix. Thanks!

[rnyak]

@sankalp-acl did you get your issue solved?

[julian-risch]

Let's continue this discussion in the issue here: #1801