Using Tools

[daaffalbari]

Why my tools always using video_id instead of transcript_text from tool_get_transcript tools.

Here is the code:

from langchain.tools import Tool, StructuredTool


tool_search_youtube = Tool(
    name="search_youtube",
    func=search_youtube,
    description="Search YouTube for a video based on a youtube_search_query",
)

tool_get_youtube_description = Tool(
    name="get_youtube_description",
    func=get_youtube_description,
    description="Get the description of a YouTube video by its video ID",
)

tool_get_transcript = Tool(
    name="get_transcript",
    func=get_transcript,
    description="Get the transcript of a YouTube video by its video ID",
)

tool_summarize_video_transcript = Tool(
    name="summarize_video_transcript",
    func=summarize_video_transcript,
    description="Summarize a YouTube video transcript from tool get_transcript",
)

tool_generate_question = Tool(
    name="generate_question",
    func=generate_question,
    description="Generate a question from a YouTube video transcript",
)

tools = [
    tool_search_youtube,
    tool_get_youtube_description,
    tool_get_transcript,
    tool_summarize_video_transcript,
    tool_generate_question,
]

from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

main_prompt = """You are a helpfull asisstant that can help to generate a course content. Generate a question from a video transcript, summarize a video transcript from get_transcript, search a video on youtube and get the description of a video. 

To get the information about that content, You can use the following tools:
1. tool_search_youtube
2. tool_get_youtube_description
3. tool_get_transcript
4. tool_summarize_video_transcript
5. tool_generate_question

Output must be like this:
{{
  "chapter_title": "",
  "summary": "",
  "concept_check": {{
    "question": "",
    "answer": [
      {{
        "option1": "",
        "option2": "",
        "option3": "",
        "option4": "",
        "true_answer": ""
      }}
    ]
  }},
  "youtube_link": "https://www.youtube.com/watch?v=<add_youtube_id_here>"
}}

"""


prompt = ChatPromptTemplate.from_messages(
    [
        ("system", main_prompt),
        ("human", "{input}"),
        # MessagesPlaceholder('chat_history', optional=True),
        MessagesPlaceholder("agent_scratchpad"),
    ]
)

from langchain.agents import AgentExecutor, create_openai_tools_agent

agent = create_openai_tools_agent(llm, tools=tools, prompt=prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

response = agent_executor.invoke({"input": "create content with chapter title cpm is Generative AI? and youtube query 'Generative AI tutorial' "})

When running summarize_video_transcript the input always video_id

[dosubot[bot]]

Hey there, @daaffalbari! I'm here to help you out with any bugs, questions, or anything else you need. Let's get that issue sorted out together!

The issue you're encountering might be due to how the tools are being invoked and how their outputs are being passed to subsequent tools. In LangChain, tool chaining is typically managed by ensuring that the output of one tool is correctly formatted and passed as input to the next tool. Here are a few things to check and consider:

1. Tool Output Handling: Ensure that the get_transcript tool is correctly returning the transcript text and not just the video ID. The function get_transcript should be designed to output the transcript text directly.

2. Tool Invocation Logic: Verify that the logic in your agent or prompt is correctly mapping the output of get_transcript to the input of summarize_video_transcript. This involves ensuring that the JSON structure or any intermediate data handling is correctly passing the transcript text.

3. Prompt and Agent Configuration: Check the configuration of your ChatPromptTemplate and AgentExecutor. Make sure that the prompt is designed to guide the model to use the transcript text as input for the summarize_video_transcript tool. The prompt should clearly specify the expected input and output for each tool.

4. Debugging and Logging: Use logging or print statements to debug the outputs at each step. This can help you trace where the video ID is being used instead of the transcript text.

5. Tool Function Definitions: Double-check the function definitions for get_transcript and summarize_video_transcript to ensure they are implemented as expected. The get_transcript function should return the transcript text, and summarize_video_transcript should accept this text as input.

By ensuring that each tool's output is correctly formatted and passed to the next tool, you should be able to resolve the issue of the summarize_video_transcript tool receiving the video ID instead of the transcript text.

To continue talking to Dosu, mention @dosu.

[daaffalbari]

@dosu can you put the code here?

[dosubot[bot]]

I can't provide the exact code for your specific setup, but I can guide you on how to ensure that the get_transcript tool outputs the transcript text correctly and that this output is used as input for the summarize_video_transcript tool.

1. Ensure get_transcript Outputs Transcript Text: Make sure the get_transcript function is returning the transcript in the desired format. Based on the LangChain framework, the get_transcript function can output the transcript in various formats like TEXT, SENTENCES, PARAGRAPHS, etc. Ensure it is set to return the full text if that's what you need ^[1].

2. Modify Tool Functions: Ensure that the summarize_video_transcript function is designed to accept the transcript text as input. You might need to adjust the function signature to accept a string of text rather than a video ID.

3. Check Tool Invocation: In your agent setup, ensure that the output from get_transcript is correctly passed to summarize_video_transcript. This might involve modifying how the tools are invoked in the AgentExecutor to ensure the correct data flow.

4. Debugging: Add print statements or logging in your tool functions to verify the data being passed between them. This can help identify where the video ID is being used instead of the transcript text.

By following these steps, you should be able to adjust your code to ensure the correct data is being passed between tools.

To continue talking to Dosu, mention @dosu.

[daaffalbari]

@dosu
i give you the full code:

def search_youtube(youtube_query: str) -> str:
    """Search YouTube for a video based on a youtube_query"""
    search_query_encoded = urllib.parse.quote(youtube_query)
    api_key = os.getenv("YOUTUBE_API_KEY")
    url = (
        f"https://www.googleapis.com/youtube/v3/search?key={api_key}"
        f"&q={search_query_encoded}&videoDuration=medium&videoEmbeddable=true"
        f"&type=video&maxResults=5&videoCaption=closedCaption"
    )

    try:
        response = requests.get(url)
        response.raise_for_status()
    except requests.exceptions.RequestException as e:
        print(f"YouTube search failed: {e}")
        return None

    data = response.json()

    if not data.get("items") or len(data["items"]) == 0:
        print("YouTube search failed")
        return None

    return f"Youtube video ID: {data['items'][0]['id']['videoId']}"


def get_youtube_description(video_id: str) -> str:
    """Get the description of a YouTube video by its video ID."""
    youtube_api_key = os.getenv("YOUTUBE_API_KEY")
    url = f"https://www.googleapis.com/youtube/v3/videos?key={youtube_api_key}&part=snippet&id={video_id}"
    try:
        response = requests.get(url)
        response.raise_for_status()
        data = response.json()
        
        return f"Youtube video description: {data['items'][0]['snippet']['description']}"
    
    except requests.exceptions.RequestException as e:
        print(f"An error occurred: {e}")
        return "No description found"    
    

def get_transcript(video_id: str) -> str:
    """Get the transcript of a YouTube video by its video ID."""
    try:
        transcript_list = YouTubeTranscriptApi.get_transcript(video_id, languages=['en'])
        transcript = ' '.join([line['text'] for line in transcript_list])
        return transcript.replace('\n', ' ')
    except (TranscriptsDisabled, NoTranscriptFound) as error:
        print('failed to fetch transcript, fetching description instead')
        return get_youtube_description(video_id)
    
    
def summarize_video_transcript(transcript_video: str) -> str:
    """Summarize a YouTube video transcript. Use video transcript as input not video_id"""
    llm = ChatOpenAI(model='gpt-4o', temperature=0)

    template_prompt = """You are an AI capable of summarizing a YouTube transcript. Make sure you return the summary, you can't return blank. IF you cannot summarize the transcript, return the first 150 words of the transcript. Summarize in 150 words or less and do not talk of the sponsors or anything unrelated to the main topic, also do not introduce what the summary is about. IF you cannot summarize the transcript, return the first 150 words of the transcript
    
    Transcript: {transcript}

    using tools get_transcript for get the transcript

    """

    prompt = ChatPromptTemplate.from_template(template_prompt)

    parser = StrOutputParser()

    chain = prompt | llm | parser

    response = chain.invoke({
        'transcript': transcript_video
    })

    return response

def generate_question(transcript_video: str) -> dict:
    """Generate a question from a YouTube video transcript."""
    llm = ChatOpenAI(model='gpt-4o', temperature=0)

    template_prompt = """
    You are an advanced AI that generates challenging multiple-choice questions designed to assess **Higher Order Thinking Skills (HOTS)**. These questions should focus on application, analysis, evaluation, or creation, rather than simple recall or understanding.
    
    Instructions:
    1. Each question must challenge the user's ability to think critically or apply concepts creatively.
    2. The correct answer must not exceed 15 words and should appear in a randomized position.
    3. Provide four options, including one correct answer and three plausible distractors.
    4. Include an explanation for the correct answer to clarify the reasoning.

    Generate one hard MCQ question based on the following context/transcript video: {transcript}.

    Return the result in JSON format like the example below, ensuring the correct answer is randomly shuffled:
    {{
      "quiz": [
        {{
          "question": "What is the capital of France?",
          "options": [
            "Paris",
            "Berlin",
            "Rome",
            "Madrid"
          ],
          "answer": "Paris"
        }},
        {{
          "question": "Which programming language is primarily used for data analysis?",
          "options": [
            "JavaScript",
            "Java",
            "C++",
            "Python"
          ],
          "answer": "Python"
        }}
      ]
    }}
    Shuffle the answer order each time. Generate 1 - 3 Questions.
    """

    prompt = ChatPromptTemplate.from_template(template_prompt)

    parser = JsonOutputParser()

    chain = prompt | llm | parser

    response = chain.invoke({
        "transcript": transcript_video
    })

    return response


tools = [
    tool_search_youtube,
    tool_get_youtube_description,
    tool_get_transcript,
    tool_summarize_video_transcript,
    tool_generate_question,
]


from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

main_prompt = """You are a helpfull asisstant that can help to generate a course content. Generate a question from a video transcript, summarize a video transcript from get_transcript, search a video on youtube and get the description of a video. 

To get the information about that content, You can use the following tools:
1. tool_search_youtube
2. tool_get_youtube_description
3. tool_get_transcript
4. tool_summarize_video_transcript
5. tool_generate_question

Output must be like this:
{{
  "chapter_title": "",
  "summary": "",
  "concept_check": {{
    "question": "",
    "answer": [
      {{
        "option1": "",
        "option2": "",
        "option3": "",
        "option4": "",
        "true_answer": ""
      }}
    ]
  }},
  "youtube_link": "https://www.youtube.com/watch?v=<add_youtube_id_here>"
}}

"""

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", main_prompt),
        ("human", "{input}"),
        # MessagesPlaceholder('chat_history', optional=True),
        MessagesPlaceholder("agent_scratchpad"),
    ]
)


from langchain.agents import AgentExecutor, create_openai_tools_agent

agent = create_openai_tools_agent(llm, tools=tools, prompt=prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)


response = agent_executor.invoke({"input": "create content with chapter title cpm is Generative AI? and youtube query 'Generative AI tutorial' "})

output:
Invoking: summarize_video_transcript with qYNweeDHiyU

why not summarize_video_transcript with output from get_transcript?

[daaffalbari]

@dosu
@dosu
i give you the full code:

def search_youtube(youtube_query: str) -> str:
    """Search YouTube for a video based on a youtube_query"""
    search_query_encoded = urllib.parse.quote(youtube_query)
    api_key = os.getenv("YOUTUBE_API_KEY")
    url = (
        f"https://www.googleapis.com/youtube/v3/search?key={api_key}"
        f"&q={search_query_encoded}&videoDuration=medium&videoEmbeddable=true"
        f"&type=video&maxResults=5&videoCaption=closedCaption"
    )

    try:
        response = requests.get(url)
        response.raise_for_status()
    except requests.exceptions.RequestException as e:
        print(f"YouTube search failed: {e}")
        return None

    data = response.json()

    if not data.get("items") or len(data["items"]) == 0:
        print("YouTube search failed")
        return None

    return f"Youtube video ID: {data['items'][0]['id']['videoId']}"


def get_youtube_description(video_id: str) -> str:
    """Get the description of a YouTube video by its video ID."""
    youtube_api_key = os.getenv("YOUTUBE_API_KEY")
    url = f"https://www.googleapis.com/youtube/v3/videos?key={youtube_api_key}&part=snippet&id={video_id}"
    try:
        response = requests.get(url)
        response.raise_for_status()
        data = response.json()
        
        return f"Youtube video description: {data['items'][0]['snippet']['description']}"
    
    except requests.exceptions.RequestException as e:
        print(f"An error occurred: {e}")
        return "No description found"    
    

def get_transcript(video_id: str) -> str:
    """Get the transcript of a YouTube video by its video ID."""
    try:
        transcript_list = YouTubeTranscriptApi.get_transcript(video_id, languages=['en'])
        transcript = ' '.join([line['text'] for line in transcript_list])
        return transcript.replace('\n', ' ')
    except (TranscriptsDisabled, NoTranscriptFound) as error:
        print('failed to fetch transcript, fetching description instead')
        return get_youtube_description(video_id)
    
    
def summarize_video_transcript(transcript_video: str) -> str:
    """Summarize a YouTube video transcript. Use video transcript as input not video_id"""
    llm = ChatOpenAI(model='gpt-4o', temperature=0)

    template_prompt = """You are an AI capable of summarizing a YouTube transcript. Make sure you return the summary, you can't return blank. IF you cannot summarize the transcript, return the first 150 words of the transcript. Summarize in 150 words or less and do not talk of the sponsors or anything unrelated to the main topic, also do not introduce what the summary is about. IF you cannot summarize the transcript, return the first 150 words of the transcript
    
    Transcript: {transcript}

    using tools get_transcript for get the transcript

    """

    prompt = ChatPromptTemplate.from_template(template_prompt)

    parser = StrOutputParser()

    chain = prompt | llm | parser

    response = chain.invoke({
        'transcript': transcript_video
    })

    return response

def generate_question(transcript_video: str) -> dict:
    """Generate a question from a YouTube video transcript."""
    llm = ChatOpenAI(model='gpt-4o', temperature=0)

    template_prompt = """
    You are an advanced AI that generates challenging multiple-choice questions designed to assess **Higher Order Thinking Skills (HOTS)**. These questions should focus on application, analysis, evaluation, or creation, rather than simple recall or understanding.
    
    Instructions:
    1. Each question must challenge the user's ability to think critically or apply concepts creatively.
    2. The correct answer must not exceed 15 words and should appear in a randomized position.
    3. Provide four options, including one correct answer and three plausible distractors.
    4. Include an explanation for the correct answer to clarify the reasoning.

    Generate one hard MCQ question based on the following context/transcript video: {transcript}.

    Return the result in JSON format like the example below, ensuring the correct answer is randomly shuffled:
    {{
      "quiz": [
        {{
          "question": "What is the capital of France?",
          "options": [
            "Paris",
            "Berlin",
            "Rome",
            "Madrid"
          ],
          "answer": "Paris"
        }},
        {{
          "question": "Which programming language is primarily used for data analysis?",
          "options": [
            "JavaScript",
            "Java",
            "C++",
            "Python"
          ],
          "answer": "Python"
        }}
      ]
    }}
    Shuffle the answer order each time. Generate 1 - 3 Questions.
    """

    prompt = ChatPromptTemplate.from_template(template_prompt)

    parser = JsonOutputParser()

    chain = prompt | llm | parser

    response = chain.invoke({
        "transcript": transcript_video
    })

    return response


tools = [
    tool_search_youtube,
    tool_get_youtube_description,
    tool_get_transcript,
    tool_summarize_video_transcript,
    tool_generate_question,
]


from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

main_prompt = """You are a helpfull asisstant that can help to generate a course content. Generate a question from a video transcript, summarize a video transcript from get_transcript, search a video on youtube and get the description of a video. 

To get the information about that content, You can use the following tools:
1. tool_search_youtube
2. tool_get_youtube_description
3. tool_get_transcript
4. tool_summarize_video_transcript
5. tool_generate_question

Output must be like this:
{{
  "chapter_title": "",
  "summary": "",
  "concept_check": {{
    "question": "",
    "answer": [
      {{
        "option1": "",
        "option2": "",
        "option3": "",
        "option4": "",
        "true_answer": ""
      }}
    ]
  }},
  "youtube_link": "https://www.youtube.com/watch?v=<add_youtube_id_here>"
}}

"""

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", main_prompt),
        ("human", "{input}"),
        # MessagesPlaceholder('chat_history', optional=True),
        MessagesPlaceholder("agent_scratchpad"),
    ]
)


from langchain.agents import AgentExecutor, create_openai_tools_agent

agent = create_openai_tools_agent(llm, tools=tools, prompt=prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)


response = agent_executor.invoke({"input": "create content with chapter title cpm is Generative AI? and youtube query 'Generative AI tutorial' "})

output:
Invoking: summarize_video_transcript with qYNweeDHiyU

why not summarize_video_transcript with output from get_transcript?

[dosubot[bot]]

The issue you're facing seems to be related to how the outputs from one tool are being passed to the next in your agent setup. Specifically, the summarize_video_transcript tool is being invoked with a video ID instead of the transcript text. Here are a few steps to address this:

1. Check Tool Output Handling: Ensure that the get_transcript function is correctly returning the transcript text. Your function seems to be doing this correctly, as it returns the transcript as a string.

2. Agent Logic and Tool Chaining: The AgentExecutor should be configured to pass the output of get_transcript (the transcript text) as the input to summarize_video_transcript. This involves ensuring that the agent's logic correctly maps the output of one tool to the input of the next.

3. Prompt and Agent Configuration: Verify that your prompt and agent setup are guiding the model to use the transcript text as input for the summarize_video_transcript tool. The prompt should clearly specify the expected input and output for each tool.

4. Debugging: Add logging or print statements to check the outputs at each step. This can help you trace where the video ID is being used instead of the transcript text.

5. Review Agent Execution Flow: Ensure that the AgentExecutor is correctly interpreting the sequence of tool invocations. The AgentExecutor should be set up to handle the flow of data between tools, ensuring that the output of get_transcript is used as the input for summarize_video_transcript.

By ensuring that each tool's output is correctly formatted and passed to the next tool, you should be able to resolve the issue of the summarize_video_transcript tool receiving the video ID instead of the transcript text.

To continue talking to Dosu, mention @dosu.