Custom our tokenizer in retriever

[sinaayyy]

I wanted to know if anyone knew how to modify the tokenizer used in the retriever model? Because in the settings of the retriever there is not one that corresponds to the tokenizer.

[bogdankostic]

Hi @sinaayyy! Can you maybe specify which retriever and model you want to use?

[sinaayyy]

Okay thank you very much for your help !

[sinaayyy]

I use your changes about autokenizer instead of DPRtokenizer but I still have a problem when I use :
retriever_dense = DensePassageRetriever( document_store=document_store_Weaviate, query_embedding_model="etalab-ia/dpr-question_encoder-fr_qa-camembert", passage_embedding_model="etalab-ia/dpr-ctx_encoder-fr_qa-camembert", )
I have this error :
You are using a Dense Passage Retriever model with the dot function. We recommend you use dot_product instead. This can be set when initializing the DocumentStore Using a model of type 'camembert' which might be incompatible with DPR encoders. Only Bert-based encoders are supported. They need input_ids, token_type_ids, attention_mask as input tensors. Using a model of type 'camembert' which might be incompatible with DPR encoders. Only Bert-based encoders are supported. They need input_ids, token_type_ids, attention_mask as input tensors.

[bogdankostic]

Is the model still running? I think you can ignore this warning message.

[sinaayyy]

I have warning but also error.
for warning :
You are using a Dense Passage Retriever model with the dot function. We recommend you use dot_product instead. This can be set when initializing the DocumentStore Using a model of type 'camembert' which might be incompatible with DPR encoders. Only Bert-based encoders are supported. They need input_ids, token_type_ids, attention_mask as input tensors. Using a model of type 'camembert' which might be incompatible with DPR encoders. Only Bert-based encoders are supported. They need input_ids, token_type_ids, attention_mask as input tensors.
for error :
WeaviateDocumentStoreError: Query results contain errors: [{'locations': [{'column': 6, 'line': 1}], 'message': 'explorer: list class: search: invalid pagination params: query maximum results exceeded', 'path': ['Get', 'Document']}]

[sinaayyy]

do you have this error when you have solve the issue ?