How can I dynamically pass grammar (guidance) to an LLM

[hadifar]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_huggingface import HuggingFaceEndpoint

llm_base = HuggingFaceEndpoint(
    endpoint_url=ENDPOINT_URL,
    max_new_tokens=1024,
    temperature=0.1,
    seed=60,
).configurable_fields(
    temperature=ConfigurableField(
        id="llm_temperature",
        name="LLM Temperature",
        description="The temperature of the LLM",
    )
)

print(llm_base.invoke("pick a random number"))
print('-' * 50)
print(llm_base.with_config(configurable={"llm_temperature": 0.9}).invoke("pick a random number"))

# The following snippet is the thing I like to implment:
#
# llm_with_grammar1 = prompt1 | llm_base.with_config(grammar1) | StrOutputParser()
# llm_with_grammar2 = prompt2 | llm_base.with_config(grammar2) | StrOutputParser()

Description

I'm trying to pass grammar dynamically to the HuggingfaceEndpoint. So far, the cleanest way that I found was using configuration, however the above code gives me an error like:

pydantic.v1.error_wrappers.ValidationError: 1 validation error for HuggingFaceEndpoint
__root__
  Please specify either an `endpoint_url` OR a `repo_id`, not both. (type=value_error)

System Info

System Information

OS: Windows
OS Version: 10.0.19045
Python Version: 3.10.10 (tags/v3.10.10:aad5f6a, Feb 7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)]

Package Information

langchain_core: 0.2.29
langchain: 0.1.13
langchain_community: 0.0.29
langsmith: 0.1.93
langchain_experimental: 0.0.55
langchain_huggingface: 0.0.3
langchain_nomic: 0.1.2
langchain_ollama: 0.1.0
langchain_openai: 0.1.20
langchain_text_splitters: 0.0.2
langchainhub: 0.1.20
langgraph: 0.2.2

[dosubot[bot]]

To dynamically pass grammar to a HuggingFaceEndpoint in LangChain without encountering a validation error, you can use the model_kwargs parameter to include any additional parameters that are not explicitly defined in the class. Here is an example of how you can do this:

from langchain_community.llms.huggingface_endpoint import HuggingFaceEndpoint

# Define your grammar or any other additional parameters
grammar1 = {
    "some_grammar_key": "some_grammar_value1"
}
grammar2 = {
    "some_grammar_key": "some_grammar_value2"
}

# Initialize the HuggingFaceEndpoint with model_kwargs
llm_base = HuggingFaceEndpoint(
    endpoint_url="http://localhost:8010/",
    max_new_tokens=1024,
    temperature=0.1,
    seed=60,
    model_kwargs=grammar1  # Pass the initial grammar here
).configurable_fields(
    temperature=ConfigurableField(
        id="llm_temperature",
        name="LLM Temperature",
        description="The temperature of the LLM",
    )
)

print(llm_base.invoke("pick a random number"))
print('-' * 50)

# Update the grammar dynamically using with_config
llm_with_grammar1 = llm_base.with_config(model_kwargs=grammar1)
llm_with_grammar2 = llm_base.with_config(model_kwargs=grammar2)

print(llm_with_grammar1.invoke("pick a random number"))
print('-' * 50)
print(llm_with_grammar2.invoke("pick a random number"))

In this example, the grammar1 and grammar2 dictionaries are passed to the model_kwargs parameter, which allows you to dynamically include additional parameters without encountering a validation error. The HuggingFaceEndpoint class is designed to handle extra parameters by transferring them to model_kwargs and ensuring they are not duplicated or invalid ^[1][2].

To continue talking to Dosu, mention @dosu.

[hadifar]

I found a solution as follow:

chat_model = ChatHuggingFace(llm=llm_base)
grammar = {"type": "json", "value": QuestionValidator.schema()}

res = chat_model.invoke([
    ("system", system_prompt),
    ("user", "{user_question}")
],
    response_format=grammar
)

How can I make the above code cleaner? E.g., how can I chain chat_model with ChatPromptTemplate?

prompt = ChatPromptTemplate([
    ("system", system_prompt),
    ("user", "{user_question}")
])

new_model = prompt | chat_model

[dosubot[bot]]

To make the code cleaner and chain chat_model with ChatPromptTemplate, you can directly use the | operator to chain the prompt template with the model. Here is the cleaned-up version of your code:

from langchain_core.prompts import ChatPromptTemplate
from langchain_community.chat_models import ChatHuggingFace

# Initialize the HuggingFaceEndpoint
llm_base = HuggingFaceEndpoint(
    endpoint_url=ENDPOINT_URL,
    max_new_tokens=1024,
    temperature=0.1,
    seed=60,
)

# Initialize the ChatHuggingFace model
chat_model = ChatHuggingFace(llm=llm_base)

# Define the grammar
grammar = {"type": "json", "value": QuestionValidator.schema()}

# Create the prompt template
prompt = ChatPromptTemplate([
    ("system", system_prompt),
    ("user", "{user_question}")
])

# Chain the prompt template with the model
chain = prompt | chat_model

# Invoke the chain with the response format
result = chain.invoke(
    {
        "user_question": "What is your name?"
    },
    response_format=grammar
)

print(result.content)

This code snippet demonstrates how to chain the ChatPromptTemplate with the ChatHuggingFace model in a clean and efficient manner. The ChatPromptTemplate class allows for flexible templated prompts for chat models, making it easy to format and invoke messages ^[1][2].

To continue talking to Dosu, mention @dosu.

[hadifar]

We have the following error:

Traceback (most recent call last):
  File "main.py", line 38, in <module>
    result = chain.invoke(
  File "\venv\lib\site-packages\langchain_core\runnables\base.py", line 2876, in invoke
    input = context.run(step.invoke, input, config, **kwargs)
TypeError: BasePromptTemplate.invoke() got an unexpected keyword argument 'response_format'

[dosubot[bot]]

To resolve the TypeError: BasePromptTemplate.invoke() got an unexpected keyword argument 'response_format' error, you need to ensure that the response_format argument is not being passed to the invoke method of BasePromptTemplate. The ChatHuggingFace class does not support a response_format argument in its methods.

Here's how you can modify your code:

1. Remove the response_format argument from the invoke method call.
2. Handle the response format separately after receiving the result.

Here is the updated code:

from langchain_core.prompts import ChatPromptTemplate
from langchain_community.chat_models import ChatHuggingFace
from langchain_community.llms.huggingface_endpoint import HuggingFaceEndpoint

# Initialize the HuggingFaceEndpoint
llm_base = HuggingFaceEndpoint(
    endpoint_url=ENDPOINT_URL,
    max_new_tokens=1024,
    temperature=0.1,
    seed=60,
)

# Initialize the ChatHuggingFace model
chat_model = ChatHuggingFace(llm=llm_base)

# Define the grammar
grammar = {"type": "json", "value": QuestionValidator.schema()}

# Create the prompt template
prompt = ChatPromptTemplate([
    ("system", system_prompt),
    ("user", "{user_question}")
])

# Chain the prompt template with the model
chain = prompt | chat_model

# Invoke the chain without the response_format argument
result = chain.invoke(
    {
        "user_question": "What is your name?"
    }
)

# Handle the response format separately
formatted_result = format_response(result, format=grammar)

print(formatted_result.content)

In this code, the invoke method is called without the response_format argument, and the response formatting is handled separately after receiving the result ^[1][2].

To continue talking to Dosu, mention @dosu.

[hadifar]

response_format must pass directly to chat model

[dosubot[bot]]

To pass the response_format directly to the ChatHuggingFace chat model, you can include it in the payload dictionary. Here is an example of how you can do this:

from langchain_core.prompts import ChatPromptTemplate
from langchain_community.chat_models import ChatHuggingFace
from langchain_community.llms.huggingface_endpoint import HuggingFaceEndpoint

# Initialize the HuggingFaceEndpoint
llm_base = HuggingFaceEndpoint(
    endpoint_url=ENDPOINT_URL,
    max_new_tokens=1024,
    temperature=0.1,
    seed=60,
)

# Initialize the ChatHuggingFace model
chat_model = ChatHuggingFace(llm=llm_base)

# Define the grammar
grammar = {"type": "json", "value": QuestionValidator.schema()}

# Create the prompt template
prompt = ChatPromptTemplate([
    ("system", system_prompt),
    ("user", "{user_question}")
])

# Chain the prompt template with the model
chain = prompt | chat_model

# Prepare the payload with response_format
payload = {
    "input": {
        "user_question": "What is your name?"
    },
    "response_format": grammar
}

# Invoke the chain with the payload
result = chain.invoke(**payload)

print(result.content)

In this code, the response_format is included in the payload dictionary and passed directly to the ChatHuggingFace model during the invocation ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[hadifar]

Traceback (most recent call last):
File "main.py", line 38, in
result = chain.invoke(
File "\venv\lib\site-packages\langchain_core\runnables\base.py", line 2876, in invoke
input = context.run(step.invoke, input, config, **kwargs)
TypeError: BasePromptTemplate.invoke() got an unexpected keyword argument 'response_format'