AzureAIChatCompletionsModel tool calling

[edisimo]

Hello!

I am using langchain for my masters thesis of controlling a quadruped robot via LLMs. First of all I have to say that langchain has without a doubt saved me an incredible amount of time and is an amazing tool.

I am able to use it for tool calling through Azure OpenAI models like 4o and so on.

I have tried to implement Llama-3-3-70B-Instruct deployed in Azure AI Foundry. It works, however I can't get the model to use any tools even when specifically asked. I have tried bind_tools() just like in the previous model, but it doesn't seem to work anyway. Could someone please help me figure this out?

Thank you very much

This is my current implementation of the model

#!/usr/bin/env python3.9

import os
import dotenv
from langchain_openai import AzureChatOpenAI
from langchain_azure_ai.chat_models import AzureAIChatCompletionsModel
from artaban_tools import move_robot

llm_streaming = False

def get_llm(model_type: str = "llama", streaming: bool = False):
    """A helper function to get the LLM instance.

    Args:
        model_type (str): Type of model to use ("openai" for Azure OpenAI GPT, "llama" for Azure ML).
        streaming (bool): Whether to enable streaming output.

    Returns:
        An instance of the selected Azure-based LLM.
    """
    global llm_streaming
    dotenv.load_dotenv(dotenv.find_dotenv())

    if model_type == "openai":
        # Use Azure OpenAI (GPT-4, GPT-3.5)
        llm_streaming = streaming
        return AzureChatOpenAI(
            api_version=os.getenv("AZURE_API_VERSION", "2024-02-15-preview"),
            azure_deployment=os.getenv("DEPLOYMENT_ID"),
            openai_api_key=os.getenv("AZURE_OPENAI_API_KEY"),
            azure_endpoint=os.getenv("API_ENDPOINT"),
            streaming=streaming,
            openai_api_type="azure",
        )

    
    elif model_type == "llama":
        # Use Azure Machine Learning (ML) deployment for LLaMA or any other model
        llm_streaming = False
        return AzureAIChatCompletionsModel(
            api_version=os.getenv("AZURE_API_VERSION", "2024-02-15-preview"),
            endpoint=os.getenv("AZURE_ML_ENDPOINT"),
            credential=os.getenv("AZURE_ML_API_KEY"),
            model_name=os.getenv("LLAMA_DEPLOYMENT_ID"),
        )
    
    else:
        raise ValueError(f"Unsupported model type: {model_type}. Choose 'openai' or 'llama'.")
    
def get_streaming():
    """Get the current streaming setting."""
    global llm_streaming
    return llm_streaming

def main():

    # Select model type (default to "openai")
    model_type = os.getenv("LLM_MODEL_TYPE", "llama")

    # Get the LLM instance
    llm = get_llm(model_type)
    if llm is None:
        raise ValueError("LLM instance is None. Check API keys and deployment settings.")

    # Send a test message
    try:
        messages = [
            {"role": "system", "content": "You are a helpful AI assistant."},
            {"role": "user", "content": "Can you use the tool move the robot for demonstration to move the robot forward?"},
        ]
        llm.bind_tools([move_robot])
        ai_msg = llm.invoke(messages)
        print(f"AI response: {ai_msg.content}")
    except Exception as e:
        print(f"Error during LLM invocation: {e}")

if __name__ == "__main__":
    main()

As for the tool, we can use a hypothetical dummy function that illustrates the point

@tool
def move_robot(linear_x: float, linear_y: float, angular_z: float) -> str:

    Move the robot based on the provided linear and angular velocities.
    All values must be within the interval <-1, 1>.

    :param linear_x: The linear velocity in the x-direction (Positive numbers move robot forward).
    :param linear_y: The linear velocity in the y-direction (Positive numbers move robot to the left).
    :param angular_z: The angular velocity around the z-axis (Positive numbers rotate the robot counterclockwise).
    :return: A success message.
   
    print(f"Moving robot with linear_x={linear_x}, linear_y={linear_y}, angular_z={angular_z}")

    return f"Success"

[dosubot[bot]]

Hello, @edisimo! 👋 I'm here to help you with any bugs, questions, or contributions you have.

I found a couple of similar discussions that might be relevant to your issue:

1. Using Model from Ollama in ChatOpenAI doesn't invoke the tools with bind_tools - This discussion is still open and might provide insights into similar tool invocation issues with different models ^[1].

2. NotImplementedError for bind_tools method in AzureMLChatOnlineEndpoint class - This is another open discussion that might be related to your problem with tool binding in Azure ML models ^[2].

Regarding your specific issue, ensure that the AzureAIChatCompletionsModel class supports tool binding and invocation. According to the LangChain documentation, this class does support tool calling ^[3]. However, there might be specific configurations or limitations that are not well-documented.

Here are a few suggestions to troubleshoot the issue:

1. Check Compatibility: Ensure that the AzureAIChatCompletionsModel is indeed compatible with the bind_tools() method. Although it is supposed to support tool calling, there might be specific requirements or configurations needed.

2. Tool Definition: Verify that the move_robot tool is correctly defined and that its schema is compatible with the model's expectations. Ensure that the tool's description and parameters are clear and concise.

3. Error Handling: Implement error handling to capture any specific exceptions or errors that occur during tool invocation. This might provide more insights into what might be going wrong.

4. Documentation and Examples: Review any available documentation or examples related to tool binding with Azure models in LangChain. This might provide additional context or steps that you might have missed.

If these steps do not resolve the issue, consider exploring the discussions mentioned above for any updates or solutions that might have been shared by other users facing similar problems.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[edisimo]

@dosu When i print out the class after binding tools this is the output: "bound=AzureAIChatCompletionsModel(endpoint='....', credential='...', model_name='LLaMA') kwargs={'tools': [{'type': 'function', 'function': {'name': 'move_robot', 'description': 'Move the robot based on the provided linear and angular velocities.\nAll values must be within the interval <-1, 1>.\n\n:param linear_x: The linear velocity in the x-direction (Positive numbers move robot forward).\n:param linear_y: The linear velocity in the y-direction (Positive numbers move robot to the left).\n:param angular_z: The angular velocity around the z-axis (Positive numbers rotate the robot counterclockwise).\n:return: A success message.', 'parameters': {'properties': {'linear_x': {'type': 'number'}, 'linear_y': {'type': 'number'}, 'angular_z': {'type': 'number'}}, 'required': ['linear_x', 'linear_y', 'angular_z'], 'type': 'object'}}}]} config={} config_factories=[] " however the LLM does not see the tool when asked.

[dosubot[bot]]

When using the bind_tools() method with the AzureAIChatCompletionsModel, ensure that the tool is correctly defined and that the model is configured to recognize and use the tool. Here are a few things to check:

1. Tool Schema: Make sure the tool's schema is correctly defined and matches the expected input format. The parameters should be clearly specified, as you have done with linear_x, linear_y, and angular_z.

2. Tool Invocation: Ensure that the model's prompt or input message explicitly requests the use of the tool. The model needs to understand that it should invoke the tool based on the user's request.

3. Model Configuration: Verify that there are no additional configurations required for the AzureAIChatCompletionsModel to recognize and use the bound tools. Sometimes, specific settings or flags might need to be enabled.

4. Error Handling: Implement detailed error handling to capture any exceptions or errors during tool invocation. This can provide insights into what might be going wrong.

5. Documentation and Examples: Review any available documentation or examples related to tool binding with Azure models in LangChain. This might provide additional context or steps that you might have missed.

If the issue persists, consider checking for any known limitations or issues with the bind_tools() method when used with Azure AI models in LangChain ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[edisimo]

Apparently Azure themselves don't support tool calling on models like LLaMA or DeepSeek. https://learn.microsoft.com/en-us/azure/ai-foundry/model-inference/concepts/models#meta

Bummer.

[edisimo]

Apparently Azure themselves don't support tool calling on models like LLaMA or DeepSeek. https://learn.microsoft.com/en-us/azure/ai-foundry/model-inference/concepts/models#meta

Bummer.