Q: vectorstore.add_documents: parameter chunks or full doc?

[jsmith173]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

python

Description

I'd like to use vectorstore.add_documents. This expects chunks or a list of extracted pdf documents?
Previously I'm used the 'from_documents' method which is works with chunks

System Info

Package Information

langchain_core: 0.2.36
langchain: 0.2.15
langchain_community: 0.2.14
langsmith: 0.1.106
langchain_chat: Installed. No version info available.
langchain_chat_stream: Installed. No version info available.
langchain_chroma: 0.1.3
langchain_ollama: 0.1.2
langchain_text_splitters: 0.2.2

[dosubot[bot]]

The vectorstore.add_documents method expects a list of extracted PDF documents, not chunks. The method processes the documents by extracting their text content and metadata, and then adds them to the vector store in batches ^[1][2][3].

Here's an example of how to use it:

from uuid import uuid4
from langchain_core.documents import Document

document_1 = Document(
    page_content="I had chocolate chip pancakes and scrambled eggs for breakfast this morning.",
    metadata={"source": "tweet"},
    id=1,
)

# ... (other documents)

documents = [
    document_1,
    # ... (other documents)
]
uuids = [str(uuid4()) for _ in range(len(documents))]
vector_store.add_documents(documents=documents, ids=uuids)

If you need to handle document chunks, you might want to use a ParentDocumentRetriever to split the documents into smaller chunks before adding them to the vector store. This ensures that the metadata of each chunk includes a reference to its parent document ^[4][2][5].

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

The text splitter class you should use depends on your specific requirements:

• CharacterTextSplitter: Use this if you want to split text based on a specific character or sequence of characters.
• RecursiveCharacterTextSplitter: Use this if you need to recursively split text by different characters to find one that works best.

LangChain also offers other text splitters for different use cases, such as HTMLHeaderTextSplitter, MarkdownHeaderTextSplitter, RecursiveJsonSplitter, LatexTextSplitter, PythonCodeTextSplitter, KonlpyTextSplitter, SpacyTextSplitter, NLTKTextSplitter, and TokenTextSplitter ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.

[jsmith173]

@dosu you write that add_documents will split the document (incoming parameter) to chunks but which text splitter class will be use in this way?

[dosubot[bot]]

The vectorstore.add_documents method uses the RecursiveCharacterTextSplitter class to split the document into chunks by default ^[1][2].

To continue talking to Dosu, mention @dosu.

[jsmith173]

@dosu
I'm using

loader = PyMuPDFLoader(file_path=file_path)
docs = loader.load()

Can I pass docs vectorstore.add_documents this way?

vectorstore.add_documents(docs)

This is not the way as you suggested previously

[dosubot[bot]]

Yes, you can pass docs from PyMuPDFLoader directly to vectorstore.add_documents. The add_documents method in the VectorStore class is designed to accept a list of Document objects, which is the format that PyMuPDFLoader provides ^[1][2][3].

To continue talking to Dosu, mention @dosu.