How can i use AzureAIDocumentIntelligenceLoader for excel sheet

[deepak-hl]

i am using AzureAIDocumentIntelligenceLoader, but in complex excel its not able to extract information, so what is best way to chunk it in azure document intellignece so that it will return meaning full chunks as an input to LLM for answering user query in RAG implementation, below is my code for reference

    def load_and_split_documents(loader, chunk_size=2000, chunk_overlap=100):
        document = loader.load()
        text_splitter = RecursiveCharacterTextSplitter(
            chunk_size=chunk_size, chunk_overlap=chunk_overlap
        )
        return text_splitter.split_documents(documents=document)

            loader = AzureAIDocumentIntelligenceLoader(
                api_endpoint=AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT, api_key=AZURE_DOCUMENT_INTELLIGENCE_KEY, file_path=file_path, api_model="prebuilt-layout"
            )
            texts = load_and_split_documents(loader)

[dosubot[bot]]

I couldn't find specific information on using AzureAIDocumentIntelligenceLoader with complex Excel files in the repository. However, for chunking documents effectively, you can use the RecursiveCharacterTextSplitter to split text into meaningful chunks. Here are some best practices:

1. Adjust Chunk Size and Overlap: Experiment with different chunk_size and chunk_overlap values to find the best balance for your data. A smaller chunk size with some overlap can help maintain context.

2. Customize Separators: If the default separators don't work well for your Excel data, you might need to customize them. For example, you can add specific characters or patterns that are common in your Excel files.

3. Use the Correct Length Function: Ensure that the length_function accurately measures the size of the text chunks according to your needs, typically using the len function for character count.

Here's an example of how you might adjust the RecursiveCharacterTextSplitter:

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=2000,
    chunk_overlap=100,
    separators=["\n\n", "\n", " ", ",", ";", "\t", ""],  # Customize as needed
    length_function=len,
    is_separator_regex=False,
)

These adjustments can help ensure that the text is split in a way that maintains semantic coherence and is suitable for further processing or analysis ^[1][2].

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

The mode in AzureAIDocumentIntelligenceLoader can be set to either "single" or "page". This setting determines whether the content is returned as pure text in a single document or split by page. For Excel files, the "page" mode works best as it allows you to handle each sheet or section of the Excel file separately, which is often necessary for maintaining the structure and context of the data ^[1].

To continue talking to Dosu, mention @dosu.

[deepak-hl]

is there any mode= elements in AzureAIDocumentIntelligenceLoader?
@dosu

[karunparashar-gc]

@dosu
For doc intelligence loader, in terms of excel files- is there a mode that evaluates and loads better for scattered excel pages/sheets

[dosubot[bot]]

In the AzureAIDocumentIntelligenceLoader, the available modes are "single", "page", and "markdown". Each mode affects how the content is represented:

• "single": Represents the entire document as a single content block, returning pure text for the whole document.
• "page": Represents each page of the document as a separate content block, returning pure text split by page.
• "markdown": Represents the document content in a markdown format, which is the default mode. This format can be easily used with MarkdownHeaderTextSplitter for semantic document chunking ^[1][2].

For Excel files, using the "page" mode might be more effective, especially if you have multiple sheets or scattered data, as it allows you to handle each sheet or section separately. However, specific optimizations for handling scattered Excel sheets are not detailed in the available documentation.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[karunparashar-gc]

Thanks @dosu , however here is the challenge
All the sheets in the excel are likely not structured and i used this to vectorize 3 excel docs with a total of 2000 rows distributed across multiple sheets in the single mode and it has been 2 hours since only one of the file has been vectorized.
I do understand the page mode the most appropriate one- but it does not read the file at all when i use that mode.
Thoughts?