Nv-rerankqa-mistral-4b-v3 model error with langchain NVIDIARerank API

[conaku]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

vector_store = Milvus(embedding_function=query_embedder, connection_args={"host": "10.209.219.165", "port": "32060"}, collection_name="LangChainCollection" )
retriever = vector_store.as_retriever()
reranker = NVIDIARerank(base_url="http://10.209.219.165:31893/v1",model="nvidia/nv-rerankqa-mistral-4b-v3")
reranking_retriever = ContextualCompressionRetriever(base_compressor=reranker, base_retriever=retriever)
llm = ChatNVIDIA(base_url="https://ashish-mistralai-deployment-1/v1",model="mistralai/mistral-7b-instruct-v0.3")
chain = ({"context": reranking_retriever, "question": RunnablePassthrough()}
        | prompt_template
        | llm
        | StrOutputParser()
    )
chain.invoke("Tell me something about langchain")    

Error is as follows

2024-08-12T15:15:59Z INFO: uvicorn.access - 10.209.219.165:59328 - "GET /health HTTP/1.1" 200
2024-08-12T15:15:59Z INFO: uvicorn.access - 10.209.219.165:59328 - "GET /health HTTP/1.1" 200
2024-08-12T15:16:00Z ERROR: root -  Type: 404: Unknown model. Available models are: ['nvidia/nv-rerankqa-mistral-4b-v3']
2024-08-12T15:16:00Z ERROR: root -  Type: 404: Unknown model. Available models are: ['nvidia/nv-rerankqa-mistral-4b-v3']
10.209.219.165:4828 - "POST /v1/ranking HTTP/1.1" 404
2024-08-12T15:16:00Z INFO: uvicorn.access - 10.209.219.165:4828 - "POST /v1/ranking HTTP/1.1" 404
2024-08-12T15:16:00Z INFO: uvicorn.access - 10.209.219.165:4828 - "POST /v1/ranking HTTP/1.1" 404

Description

I have deployed nv-rerankqa-mistral-4b-v3:1.0.1 model using helm chart available in NGC Catalog. Deployment has been successful. I am able to query model using curl successfully however getting error with langchain NVIDIARerank APIs

System Info

python -m langchain_core.sys_info

System Information

OS: Linux
OS Version: #1 SMP Wed Apr 5 13:35:01 EDT 2023
Python Version: 3.11.5 (main, Sep 22 2023, 15:34:29) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)]

Package Information

langchain_core: 0.2.11
langchain: 0.2.6
langchain_community: 0.2.6
langsmith: 0.1.83
langchain_milvus: 0.1.1
langchain_nvidia_ai_endpoints: 0.1.2
langchain_text_splitters: 0.2.2
langserve: 0.2.2

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langgraph

[dosubot[bot]]

To resolve the 404 error indicating an unknown model when using the nv-rerankqa-mistral-4b-v3 model with the LangChain NVIDIARerank API, ensure that you are correctly specifying the model and base URL. Here is an example of how to connect to a reranking NIM:

from langchain_nvidia_ai_endpoints import NVIDIARerank

# connect to a reranking NIM running at localhost:2016
ranker = NVIDIARerank(base_url="http://localhost:2016/v1", model="nv-rerankqa-mistral-4b-v3")

Make sure that the model name nv-rerankqa-mistral-4b-v3 is correctly specified and that the base URL is accurate. If the model is accessible via curl, double-check that the endpoint URL and model name match exactly in your LangChain code.

Additionally, ensure that you have set up your NVIDIA API key correctly. Follow these steps to set up your API key:

1. Create a free account with NVIDIA, which hosts NVIDIA AI Foundation models.
2. Select the Retrieval tab, then select your model of choice.
3. Under Input select the Python tab, and click Get API Key. Then click Generate Key.
4. Copy and save the generated key as NVIDIA_API_KEY.

You can set the API key in your environment as follows:

import getpass
import os

if os.environ.get("NVIDIA_API_KEY", "").startswith("nvapi-"):
    print("Valid NVIDIA_API_KEY already in environment. Delete to reset")
else:
    nvapi_key = getpass.getpass("NVAPI Key (starts with nvapi-): ")
    assert nvapi_key.startswith("nvapi-"), f"{nvapi_key[:5]}... is not a valid key"
    os.environ["NVIDIA_API_KEY"] = nvapi_key

By ensuring the correct model name, base URL, and API key setup, you should be able to resolve the 404 error ^[1][2].

To continue talking to Dosu, mention @dosu.