Mistral-2503 bad performance in tool calling comparing Mistral-2501

[alejandrofdzllorente]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

llm = ChatMistralAI(
    model="mistral-small-2501",
    # model="mistral-small-2503",
    api_key=os.getenv("MISTRAL_API_KEY"),
    base_url=os.getenv("MISTRAL_BASE_URL", "https://api.mistral.ai/v1"),
    model_kwargs={"tool_choice": "auto"},
    # response_format = {"type": "json_object"} #tried to. It does not work.
)


def multiply(a: int, b: int) -> int:
    """Multiply a and b.

    Args:
        a: first int
        b: second int
    """
    return a * b


def add(a: int, b: int) -> int:
    """Adds a and b.

    Args:
        a: first int
        b: second int
    """
    return a + b


def divide(a: int, b: int) -> float:
    """Divide a and b.

    Args:
        a: first int
        b: second int
    """
    return a / b


tools = [add, multiply, divide]


llm_with_tools = llm.bind_tools(tools)

# System message
sys_msg = SystemMessage(
    content="""You are a helpful assistant tasked with performing arithmetic on a set of inputs."""
)

# Node
def assistant(state: MessagesState):
    msgs = llm_with_tools.invoke([sys_msg] + state["messages"])
    try:
        print(msgs.additional_kwargs["tool_calls"])
    except:
        print(msgs)
    time.sleep(2)
    return {"messages": [msgs]}


# Graph
builder = StateGraph(MessagesState)

# Define nodes: these do the work
builder.add_node("assistant", assistant)
builder.add_node("tools", ToolNode(tools))

# Define edges: these determine how the control flow moves
builder.add_edge(START, "assistant")
builder.add_conditional_edges(
    "assistant",
    # If the latest message (result) from assistant is a tool call -> tools_condition routes to tools
    # If the latest message (result) from assistant is a not a tool call -> tools_condition routes to END
    tools_condition,
)
builder.add_edge("tools", "assistant")
react_graph = builder.compile()

# # Show
# display(Image(react_graph.get_graph(xray=True).draw_mermaid_png()))


messages = [
    HumanMessage(
        "Add 3 and 4. Multiply the output by 2. Divide the output by 5. Add 2 finally."
    )
]
messages = react_graph.invoke({"messages": messages})

for m in messages["messages"]:
    m.pretty_print()

Description

I'm trying to use tools with the new mistral-2503. but it is not taking the tools in the proper way, it takes the first tool (add) but with the rest it fails, logs here:

root@c3fba1b9eb09:/app# /usr/local/bin/python /app/langgraph_use_case/examples/tools_basic.py
[{'id': 'KU67jrBgg', 'function': {'name': 'add', 'arguments': '{"a": 3, "b": 4}'}, 'index': 0}]
content='[{"name": "multiply", "arguments": {"a": 7, "b": 2}}]' additional_kwargs={} response_metadata={'token_usage': {'prompt_tokens': 343, 'total_tokens': 367, 'completion_tokens': 24}, 'model_name': 'mistral-small-2503', 'model': 'mistral-small-2503', 'finish_reason': 'stop'} id='run-4573bae1-0fa8-4dab-9837-9f9fcb74b526-0' usage_metadata={'input_tokens': 343, 'output_tokens': 24, 'total_tokens': 367}
================================ Human Message =================================

Add 3 and 4. Multiply the output by 2. Divide the output by 5. Add 2 finally.
================================== Ai Message ==================================
Tool Calls:
  add (KU67jrBgg)
 Call ID: KU67jrBgg
  Args:
    a: 3
    b: 4
================================= Tool Message =================================
Name: add

7
================================== Ai Message ==================================

[{"name": "multiply", "arguments": {"a": 7, "b": 2}}]

But when I try the mistral-small-2501 it takes all the tools perfectly! Logs here:

root@c3fba1b9eb09:/app# /usr/local/bin/python /app/langgraph_use_case/examples/tools_basic.py
[{'id': 'zIvv9C56a', 'function': {'name': 'add', 'arguments': '{"a": 3, "b": 4}'}, 'index': 0}]
[{'id': 'vCuAedfGP', 'function': {'name': 'multiply', 'arguments': '{"a": 7, "b": 2}'}, 'index': 0}]
[{'id': '9IzufIJtZ', 'function': {'name': 'divide', 'arguments': '{"a": 14, "b": 5}'}, 'index': 0}]
[{'id': 'ppeqeHc9H', 'function': {'name': 'add', 'arguments': '{"a": 2, "b": 2}'}, 'index': 0}]
content='The final output is 4.' additional_kwargs={} response_metadata={'token_usage': {'prompt_tokens': 491, 'total_tokens': 499, 'completion_tokens': 8}, 'model_name': 'mistral-small-2501', 'model': 'mistral-small-2501', 'finish_reason': 'stop'} id='run-2e9c4ff1-c0fd-4d13-9dba-1c1b437470ce-0' usage_metadata={'input_tokens': 491, 'output_tokens': 8, 'total_tokens': 499}
================================ Human Message =================================

Add 3 and 4. Multiply the output by 2. Divide the output by 5. Add 2 finally.
================================== Ai Message ==================================
Tool Calls:
  add (zIvv9C56a)
 Call ID: zIvv9C56a
  Args:
    a: 3
    b: 4
================================= Tool Message =================================
Name: add

7
================================== Ai Message ==================================
Tool Calls:
  multiply (vCuAedfGP)
 Call ID: vCuAedfGP
  Args:
    a: 7
    b: 2
================================= Tool Message =================================
Name: multiply

14
================================== Ai Message ==================================
Tool Calls:
  divide (9IzufIJtZ)
 Call ID: 9IzufIJtZ
  Args:
    a: 14
    b: 5
================================= Tool Message =================================
Name: divide

2.8
================================== Ai Message ==================================
Tool Calls:
  add (ppeqeHc9H)
 Call ID: ppeqeHc9H
  Args:
    a: 2
    b: 2
================================= Tool Message =================================
Name: add

4
================================== Ai Message ==================================

The final output is 4.

I would love to use the new model but this problem is annoying me. I just have the latest versions of langchain and langchain_mistralai but it's still failing.

Thanks in advance.

System Info

langchain==0.3.21
langchain-chroma==0.2.2
langchain-community==0.3.17
langchain-core==0.3.49
langchain-experimental==0.3.4
langchain-mistralai==0.2.10
langchain-ollama==0.2.2
langchain-openai==0.3.6
langchain-text-splitters==0.3.7