Chain_of_tables is giving wonderful insights (I am very surprised) but it is timing out for bigger datasets via AzureOpenAI.

[mraguth]

I am using a Chain of tables for 200 row dataset (with 5 columns). However, it is timing out every time while I am using AzureOpenAI.
If I use 20 or 30 rows, it is working fine. I have bigger prompt. Is that an issue? Is there any mitigation for this process? Please help.

https://github.com/run-llama/llama-hub/blob/main/llama_hub/llama_packs/tables/chain_of_table/chain_of_table.ipynb

I am using this particular function:

from llama_index.prompts import PromptTemplate
from llama_index.query_pipeline import QueryPipeline

prompt_str = """\
Here's a serialized table.

{serialized_table}

Given this table please answer the question: {question}
Answer: """
prompt = PromptTemplate(prompt_str)
prompt_c = prompt.as_query_component(partial={"serialized_table": serialize_table(df)})

[dosubot[bot]]

Hey @mraguth, fancy seeing you here again! Diving into another data adventure, I see. Let's crack this one together.

Based on the context you've provided, it seems like the timeout issue you're experiencing with AzureOpenAI when using the Chain_of_tables feature from the LlamaIndex repository might be due to the size of your dataset. Here are a few approaches you could consider to mitigate this issue:

1. Batch Processing: You can adjust the embed_batch_size parameter in the AzureOpenAIEmbedding class constructor to a lower value that suits your dataset size and complexity. This parameter controls the size of data batches sent to AzureOpenAI for processing. By reducing the batch size, AzureOpenAI can process smaller chunks of data at a time, which could help prevent timeouts. The default value for embed_batch_size is DEFAULT_EMBED_BATCH_SIZE, which is set to 10 in the LlamaIndex codebase.

2. Increase Timeout Settings: You can increase the timeout settings for the HTTP client. While the provided code does not directly expose these settings, you can pass a custom httpx.Client instance with increased timeout settings to the AzureOpenAIEmbedding class constructor. Here's an example of how you can do this:

from httpx import Client, Timeout
from llama_index.legacy.embeddings.azure_openai import AzureOpenAIEmbedding

# Increase the timeout duration
timeout = Timeout(30.0)  # Adjust the timeout as needed
http_client = Client(timeout=timeout)

# Initialize AzureOpenAIEmbedding with the custom http_client
azure_openai_embedding = AzureOpenAIEmbedding(http_client=http_client, ...)

3. Retry Mechanism: You can adjust the max_retries parameter in the AzureOpenAIEmbedding class constructor. This parameter specifies how many times an operation should be retried in case of failure. While increasing retries does not directly mitigate timeouts, it can help recover from transient issues.

4. Asynchronous Processing: If your application's architecture allows, you could consider using asynchronous processing to handle larger datasets. The _get_aclient method in the AzureOpenAIEmbedding class suggests that asynchronous operations are supported. Processing data asynchronously can improve the overall efficiency and may help in managing timeouts better by not blocking the execution flow while waiting for responses.

Remember to test these adjustments in a controlled environment to find the optimal configuration that works for your specific dataset and use case. You can find more information about these parameters in the LlamaIndex codebase.

Sources

• llama-index-legacy/llama_index/legacy/embeddings/azure_openai.py
• llama-index-integrations/embeddings/llama-index-embeddings-azure-openai/llama_index/embeddings/azure_openai/base.py
• llama-index-core/llama_index/core/constants.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[mraguth]

Can someone help me how I add 'embed_batch_size' either in llm or embeddings configuration please? I am using AzureOpenai.