ONNXRuntime for embeddings task

[allegro-lukasz-dragan]

Hi! I'm trying to create EmbeddingRetriever using ONNX model created previously with AdaptiveModel.convert_to_onnx.
Having called

EmbeddingRetriever(embedding_model='<a path to the ONNX model dir>')

I receive:

Traceback (most recent call last):
  File "haystack-poc/main.py", line 3, in <module>
    retriever = EmbeddingRetriever(
  File "haystack/nodes/base.py", line 48, in wrapper_exportable_to_yaml
    init_func(self, *args, **kwargs)
  File "haystack/nodes/retriever/dense.py", line 1550, in __init__
    self.embedding_encoder = _EMBEDDING_ENCODERS[self.model_format](self)
  File "haystack/nodes/retriever/_embedding_encoder.py", line 92, in __init__
    self.embedding_model = Inferencer.load(
  File "haystack/modeling/infer.py", line 247, in load
    return cls(
  File "haystack/modeling/infer.py", line 105, in __init__
    self.model.language_model.extraction_layer = extraction_layer
AttributeError: 'ONNXAdaptiveModel' object has no attribute 'language_model'

It looks like ONNXAdaptiveModel is not compatible with Inferencer with embeddings task even though it is passed to Inferencer __init__ by the library itself.

Am I doing something wrong or is ONNX in embeddings task currently not supported? If so, are you planning to support it in the future? I think it is not very clear now.

[vblagoje]

Hi @allegro-lukasz-dragan , can you please create an issue for this? I am not familiar with this particular feature, but I'll take a look. Can you create an isolated minima use-case to demonstrate this (colab perhaps)?

[vblagoje]

Hey @allegro-lukasz-dragan, which model did you convert_to_onnx? Can you provide the haystack-poc/main.py snippet?

[allegro-lukasz-dragan]

I will try to prepare an example to reproduce this error and then reply

[vblagoje]

@allegro-lukasz-dragan, we don't support onnx models for retrievers only for QA prediction. For example, you can use the ONNX model in the FarmReader. Having said that, we found a bug #3210 and the fix is already in place. It should be ready for the v1.9.0 release. The FarmReader support of onnx models should also work in Haystack releases before 1.7.0

[allegro-lukasz-dragan]

Ok, thanks for the quick reply