BioGPT retriever

[ali-issa9]

Hello everyone, i would like to know how we can add the biogpt model as a retirver inside haystack? does anyone have an idea?

thank you

[anakin87]

Hello @ali-issa9! I am not an expert on biological Language models, but I'll try to give you some advice...

---

In Haystack, the retriever is a node that, given a query as input, returns the most relevant/nearest documents.
It does not provide answers directly.

Retrievers can be sparse or dense.
Sparse retrievers (such as BM25, used in Elasticsearch) are based on lexical matches, while dense retrievers use Language Models and capture the semantic similarity between query and documents.

Not all Language Model can be used in a dense retriever. We need a Language Model that can produce good vector representations of documents, such as models from the family of Sentence Transformers.

---

In contrast, BioGPT is a generative model: given a prompt, it can generate more coherent text.
It is therefore not suitable to act as an encoder and to be used in a retriever.

---

What are you trying to achieve/build?
With a better view, maybe me or someone else of the community can help!

[ali-issa9]

Hello @anakin87 , i am trying to add biogpt, because i want to store several medical paragraphs inside a document store, and given a query i want to retrieve a document for example.

I already saw that we are able to generate word embeddings using the following code :

from transformers import BioGptTokenizer, BioGptForCausalLM
tokenizer = BioGptTokenizer.from_pretrained("microsoft/biogpt")
model = BioGptForCausalLM.from_pretrained("microsoft/biogpt")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)

source : https://github.com/microsoft/BioGPT

Moreover, some of them said that we can create the embeddings using AutoModel https://huggingface.co/microsoft/biogpt/discussions/11

But also was not able to add it to my retriever.

So given these resources and links that i provided , am I missing something?

[anakin87]

Now I'm understanding better your problem...

1. With the above code, you are not generating sentence embeddings but token representations.
   In fact, output[0].shape is torch.Size([1, 12, 42384]) where 12 is the number of tokens (≈words).
   To get a sentence embedding, you need to do a pooling operation, as shown in Sentence Transformers docs.
   In any case, I doubt you can take a model trained for text generation and use it to produce embeddings that make sense, without further training.

2. Despite my doubts, I'm trying to use this model in Haystack, but it doesn't work out of the box...

from haystack.document_stores import InMemoryDocumentStore
from haystack.nodes import EmbeddingRetriever

document_store = InMemoryDocumentStore(
    embedding_dim=42384
)

retriever = EmbeddingRetriever(
    document_store=document_store,
   embedding_model="microsoft/biogpt",
   model_format="transformers"
)

gives a KeyError raised by the Transformers library

3. In my humble opinion, you should find a model that is suitable for your task.
   I've heard of BioBERT. There are also Sentence similarity models based on it which are good for retrieval.
   An example: BioBERT-mnli-snli-scinli-scitail-mednli-stsb

[ali-issa9]

Hello @anakin87, thank you for this info, when i was looking into this model BioBERT-mnli-snli-scinli-scitail-mednli-stsb, I realized that the owner of this model fine-tuned an original model from Microsoft using several datasets. So what I would like to know, is how am i able to take a pre-trained model and start fine-tuning it. Is there any general process for that? Thank you

[anakin87]

In your place I would try to make some tests using the pretrained model and, if it doesn't satisfy you, proceed with the fine-tuning.

[ali-issa9]

thank you ! @anakin87