Getting error while using confluence loader with attachments

[deepak-hl]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

confluence_url = config.get("confluence_url", None)
username = config.get("username", None)
api_key = config.get("api_key", None)
space_keys = config.get("space_key", None)  # Assuming this is a list of space keys
documents = []

embedding = OpenAIEmbeddings()

# Initialize the ConfluenceLoader with all required parameters
loader = ConfluenceLoader(
    url=confluence_url,
    username=username,
    api_key=api_key,
    limit=100,  # Set the limit here
    include_attachments=True  # Set include_attachments here
)

for space_key in space_keys:
    loader.space_key = space_key  # Set the space_key for each iteration
    documents.extend(loader.load())

text_splitter = CharacterTextSplitter(
    chunk_size=6000, chunk_overlap=50
)
texts = text_splitter.split_documents(documents)
text_splitter = TokenTextSplitter(
    chunk_size=1000, chunk_overlap=10, encoding_name="cl100k_base"
)
texts = text_splitter.split_documents(texts)

PGVector.from_texts(
    embedding=embedding,
    texts=texts,
    collection_name=persist_directory,
    connection_string=CONNECTION_STRING,
)

Description

While running confluence loader with include_attachments=True, I am getting below error:
'message'
400 Client Error: Bad Request for url: https://api.media.atlassian.com/file/UNKNOWN_MEDIA_ID/binary?token=eyJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJmMTc3YzU0Ny03Y2NlLTQyN2ItYmQ5Yy00M2I4ZmZlOWRkZDUiLCJhY2Nlc3MiOnsidXJuOmZpbGVzdG9yZTpmaWxlOlVOS05PV05fTUVESUFfSUQiOlsicmVhZCJdfSwiZXhwIjoxNzIzNzE4NzAyLCJuYmYiOjE3MjM2MzU3ODJ9.ZUaXdaEuTxWnqKcIk4rXvSPhBAhmixzWQOPxXzqBbBo&client=f177c547-7cce-427b-bd9c-43b8ffe9ddd5&name=Invalid%20file%20id%20-%20UNKNOWN_MEDIA_ID&max-age=2940

System Info

langchain
ubuntu

[dosubot[bot]]

To resolve the "400 Client Error: Bad Request" when using ConfluenceLoader with include_attachments=True, ensure that you have the latest version of the atlassian-python-api package installed. You can upgrade it using the following command:

%pip install --upgrade --quiet atlassian-python-api

Additionally, verify that the space_key and page_id values you are using are correct and exist in your Confluence instance. These values can be found in the URL of a page in Confluence:

https://yoursite.atlassian.com/wiki/spaces/<space_key>/pages/<page_id>

Here is an example of how to use ConfluenceLoader with include_attachments=True:

from langchain_community.document_loaders import ConfluenceLoader

loader = ConfluenceLoader(
    url="https://yoursite.atlassian.com/wiki", username="me", api_key="12345"
)
documents = loader.load(space_key="SPACE", include_attachments=True, limit=50)

If you are using a Personal Access Token (PAT) for an on-prem installation, the code would look like this:

from langchain_community.document_loaders import ConfluenceLoader

loader = ConfluenceLoader(
    url="https://confluence.yoursite.com/",
    token="12345",
    cloud=False,
    space_key="SPACE",
    limit=50,
)
documents = loader.load(include_attachments=True, max_pages=50)

Ensure that the user associated with the PAT has the necessary permissions to access the documents and attachments. Additionally, make sure to specify the cloud parameter correctly based on whether you are using a cloud or on-prem installation ^[1][2].

To continue talking to Dosu, mention @dosu.

[deepak-hl]

@dosu how to skip this file?

[dosubot[bot]]

To skip files that cause errors without failing the entire load process, you can modify the ConfluenceLoader to handle exceptions and continue processing other attachments. Here is an updated version of the code that skips problematic files:

from langchain_community.document_loaders import ConfluenceLoader
import logging

def load_confluence_documents(url, username=None, api_key=None, token=None, space_key=None, include_attachments=True, limit=50, max_pages=50):
    try:
        if token:
            loader = ConfluenceLoader(url=url, token=token)
        else:
            loader = ConfluenceLoader(url=url, username=username, api_key=api_key)
        
        documents = []
        attachments = loader.confluence.get_attachments_from_content(space_key)["results"]
        
        for attachment in attachments:
            media_type = attachment["metadata"]["mediaType"]
            absolute_url = loader.base_url + attachment["_links"]["download"]
            title = attachment["title"]
            try:
                if media_type == "application/pdf":
                    text = title + loader.process_pdf(absolute_url)
                elif media_type in ["image/png", "image/jpg", "image/jpeg"]:
                    text = title + loader.process_image(absolute_url)
                elif media_type == "application/vnd.openxmlformats-officedocument.wordprocessingml.document":
                    text = title + loader.process_doc(absolute_url)
                elif media_type == "application/vnd.ms-excel":
                    text = title + loader.process_xls(absolute_url)
                elif media_type == "image/svg+xml":
                    text = title + loader.process_svg(absolute_url)
                else:
                    continue
                documents.append(text)
            except requests.HTTPError as e:
                if e.response.status_code == 404:
                    logging.warning(f"Attachment not found at {absolute_url}")
                    continue
                else:
                    logging.error(f"Error processing attachment {title}: {e}")
                    continue
            except Exception as e:
                logging.error(f"Unexpected error processing attachment {title}: {e}")
                continue
        
        return documents
    except Exception as e:
        logging.error(f"An unexpected error occurred: {e}")
        return None

# Example usage
documents = load_confluence_documents(
    url="https://yoursite.atlassian.com/wiki", 
    username="me", 
    api_key="12345", 
    space_key="SPACE", 
    include_attachments=True, 
    limit=50
)

In this code, the try-except blocks within the loop handle specific errors like requests.HTTPError and general exceptions, logging the errors and continuing to the next attachment. This ensures that the loader skips problematic files and continues processing the remaining attachments ^[1].

To continue talking to Dosu, mention @dosu.

[deepak-hl]

@dosu is there any inbuit method to skip this missing file?

[dosubot[bot]]

No, there is no built-in method in ConfluenceLoader to automatically skip missing files or incorrect IDs without manually handling exceptions. The current implementation requires you to handle such cases manually within your code. For example, you can catch specific exceptions and continue processing other attachments:

try:
    if media_type == "application/pdf":
        text = title + self.process_pdf(absolute_url)
    elif media_type in ["image/png", "image/jpg", "image/jpeg"]:
        text = title + self.process_image(absolute_url)
    elif media_type == "application/vnd.openxmlformats-officedocument.wordprocessingml.document":
        text = title + self.process_doc(absolute_url)
    elif media_type == "application/vnd.ms-excel":
        text = title + self.process_xls(absolute_url)
    elif media_type == "image/svg+xml":
        text = title + self.process_svg(absolute_url)
    else:
        continue
    texts.append(text)
except requests.HTTPError as e:
    if e.response.status_code == 404:
        print(f"Attachment not found at {absolute_url}")
        continue
    else:
        raise

This code handles 404 errors by printing a message and continuing, but other errors will still raise exceptions ^[1].

To continue talking to Dosu, mention @dosu.

[MS-Kinguin]

I have the exact same problem, I followed the snippet you provided and I'm still getting:

An unexpected error occurred: Unsupported image format/type

Maybe something changed in the loader and the solution is different. I'd appreciate some tips on the matter.

@dosu