Why does my request get re-routed to mistral-nemo?

[777advait]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_huggingface import HuggingFaceEndpoint
from langchain_core.prompts import PromptTemplate
from langchain_core.output_parsers import PydanticOutputParser
import requests

from app.models.profile_model import ProfileOutput, UserData
from app.core.config import settings
from app.core.logger import log
from app.core.templates import PROFILE_GENERATION_PROMPT, USER_INPUT_TEMPLATE


class ProfileService:
    def __init__(self, model: str = "tiiuae/falcon-7b-instruct"):
        self.llm = HuggingFaceEndpoint(
            repo_id="tiiuae/falcon-7b-instruct",
            task="text-generation",
            huggingfacehub_api_token=settings.HF_ACCESS_TOKEN,
            model_kwargs={
                "use_cache": False
            },
            cache=False,
            top_p=0.9,
            temperature=0.3,
            do_sample=True,
        )

    def generate_profile(self, user_data: UserData) -> ProfileOutput:
        parser = PydanticOutputParser(pydantic_object=ProfileOutput)

        prompt = PromptTemplate(
            template=PROFILE_GENERATION_PROMPT,
            input_variables=["user_input"],
            partial_variables={
                "format_instructions": parser.get_format_instructions()}
        )

        chain = prompt | self.llm | parser

        user_input_data = user_data.model_dump()

        output = chain.invoke(
            {"user_input": USER_INPUT_TEMPLATE.format(
                user_input_data['bio'],
                user_input_data['interests'],
                user_input_data['achievements'],
                user_input_data['looking_for']
            )}
        )

        return output

    def check_model(self, model="microsoft/phi-2"):
        headers = {"Authorization": f"Bearer {settings.HF_ACCESS_TOKEN}"}

        quota = requests.get(
            "https://huggingface.co/api/whoami-v2", headers=headers)

        payload = {"inputs": "Test", "parameters": {"temperature": 0.3}}
        model_response = requests.post(
            f"https://api-inference.huggingface.co/models/{model}", headers=headers, json=payload)

        log.info(self.llm.repo_id)

        return quota.json(), model_response.json()


try:
    log.info(f"Booting up [bold blue]{__name__}[/]...", extra={"markup": True})
    profile_service = ProfileService()
    log.info(profile_service.check_model("tiiuae/falcon-7b-instruct"))
except Exception as e:
    log.error(e)
    raise RuntimeError(f"Failed to boot up {__name__}")

Description

i am using falcon-7b model but when i make a request to my service it seemingly uses mistral-nemo-instruct from what i can infer from the logs

System Info

python3 -m langchain_core.sys_info

System Information
------------------
> OS:  Darwin
> OS Version:  Darwin Kernel Version 23.6.0: Mon Jul 29 21:13:00 PDT 2024; root:xnu-10063.141.2~1/RELEASE_X86_64
> Python Version:  3.10.11 (v3.10.11:7d4cc5aa85, Apr  4 2023, 19:05:19) [Clang 13.0.0 (clang-1300.0.29.30)]

Package Information
-------------------
> langchain_core: 0.3.33
> langsmith: 0.3.3
> langchain_huggingface: 0.1.2

Optional packages not installed
-------------------------------
> langserve

Other Dependencies
------------------
> httpx: 0.28.1
> huggingface-hub: 0.28.1
> jsonpatch: 1.33
> langsmith-pyo3: Installed. No version info available.
> orjson: 3.10.15
> packaging: 24.2
> pydantic: 2.10.6
> pytest: Installed. No version info available.
> PyYAML: 6.0.2
> requests: 2.32.3
> requests-toolbelt: 1.0.0
> rich: 13.9.4
> sentence-transformers: 3.4.1
> tenacity: 9.0.0
> tokenizers: 0.21.0
> transformers: 4.48.2
> typing-extensions: 4.12.2
> zstandard: 0.23.0

[SMAntony]

#29565 This looks like a problem with huggingface_hub library. Downgrade to 0.27.0 and the issue will be fixed.