SentenceTransformersSimilarityRanker() with vLLM?

[mariosconsta]

Hello!

We are using vLLM to serve a couple of models and we are using its OpenAI compatible API keys.

For example:

query_rephrase_llm = OpenAIGenerator(api_key=Secret.from_token(llm_config['VLLM_API_KEY']),
                                     api_base_url=f"{llm_config['VLLM_LLM_URL']}/v1",
                                     model=llm_config['MODEL_NAME'],)

chat_generator = OpenAIChatGenerator(api_key=Secret.from_token(llm_config['VLLM_API_KEY']),
                                     api_base_url=f"{llm_config['VLLM_LLM_URL']}/v1",
                                     model=llm_config['MODEL_NAME'],)

text_embedder = OpenAITextEmbedder(api_key=Secret.from_token(embd_config['VLLM_API_KEY']),
                                     api_base_url=f"{embd_config['VLLM_EMBD_URL']}/v1",
                                     model=embd_config['MODEL_NAME'],)
                                     

And then we have our re-ranker as well, but I am not sure how to initialize it to talk with vLLM.

e.g.

ranker_query_prefix = "Given the following user query, determine if the information provided below addresses the inquiry accurately."
ranker = SentenceTransformersSimilarityRanker(query_prefix=ranker_query_prefix, top_k=5)
ranker.warm_up()         

I've been reading the vLLM documentation but I would like some input from someone who maybe knows this with Haystack.