How to Persist LangChain's ParentDocumentRetriever and reinitilize it at a later point for use over a Large corpus of documents.

[AkashBais]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_community.vectorstores.chroma import Chroma
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_aws.embeddings.bedrock import BedrockEmbeddings

from langchain.storage import LocalFileStore
from langchain.retrievers import ParentDocumentRetriever
from langchain.storage._lc_store import create_kv_docstore

persist_directory = "local_vectorstore"
collection_name = "datadocuments"
# PROJECT_ROOT = "...." #insert your project root directory name here
persist_directory = os.path.join(config.VECTOR_DB_PERSIST_BASE,db_name, persist_directory)

vectorstore = FAISS.from_documents(documents=splits,
                                   embedding = BedrockEmbeddings(client=self.get_bedrock_runtime_client(),
                                                             model_id=model_id,
                                                            normalize = True))



# The storage layer for the parent documents
local_store = "local_docstore"
local_store_path = os.path.join(config.VECTOR_DB_PERSIST_BASE,db_name, local_store)
local_store = LocalFileStore(local_store_path)
docstore = create_kv_docstore(local_store)

# This text splitter is used to create the child documents
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2048, chunk_overlap=200)
child_splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=50)

retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=docstore,
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)

retriever.add_documents(splits)

Description

In the above code splits is a list of langchain documents. I want to use ParentDocumentRetriever on them. My issue is how do i persist the ParentDocumentRetriever for a later use.
retriever.add_documents(splits) method creates 2 folders 'local_docstore' and 'local_vectorstore' but i can't figureout how do i use them to recreate aninstance of ParentDocumentRetriever with the added documents.

System Info

System Information

OS: Linux
OS Version: #1 SMP Wed Aug 7 16:53:32 UTC 2024
Python Version: 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:36:39) [GCC 10.4.0]

Package Information

langchain_core: 0.2.41
langchain: 0.2.16
langchain_community: 0.2.5
langsmith: 0.1.125
langchain_aws: 0.1.18
langchain_chroma: 0.1.4
langchain_text_splitters: 0.2.4
langchainhub: 0.1.21

Optional packages not installed

langgraph
langserve

Other Dependencies

aiohttp: 3.10.5
async-timeout: 4.0.3
boto3: 1.34.162
chromadb: 0.5.7
dataclasses-json: 0.6.7
fastapi: 0.110.1
httpx: 0.27.0
jsonpatch: 1.33
numpy: 1.26.4
orjson: 3.10.7
packaging: 23.2
pydantic: 2.7.0
PyYAML: 6.0.1
requests: 2.31.0
SQLAlchemy: 2.0.29
tenacity: 8.2.3
types-requests: 2.32.0.20240914
typing-extensions: 4.11.0

[dosubot[bot]]

To persist LangChain's ParentDocumentRetriever and reinitialize it at a later point, you need to save the state of the vectorstore and docstore used by the retriever. Here is an example of how you can achieve this:

1. Persisting the Retriever State:

   • Save the state of the vectorstore and docstore to disk or another persistent storage.

2. Reinitializing the Retriever:

   • Load the state of the vectorstore and docstore from the persistent storage and use them to reinitialize the ParentDocumentRetriever.

Here is a code example demonstrating these steps:

from langchain_community.vectorstores.chroma import Chroma
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_aws.embeddings.bedrock import BedrockEmbeddings
from langchain.storage import LocalFileStore
from langchain.retrievers import ParentDocumentRetriever
from langchain.storage._lc_store import create_kv_docstore

# Initialize the retriever
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2048, chunk_overlap=200)
child_splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=50)

# Assuming `splits` is your list of documents
vectorstore = FAISS.from_documents(
    documents=splits,
    embedding=BedrockEmbeddings(
        client=self.get_bedrock_runtime_client(),
        model_id=model_id,
        normalize=True
    )
)

# The storage layer for the parent documents
local_store_path = os.path.join(config.VECTOR_DB_PERSIST_BASE, db_name, "local_docstore")
local_store = LocalFileStore(local_store_path)
docstore = create_kv_docstore(local_store)

retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=docstore,
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)

retriever.add_documents(splits)

# Persist the state of vectorstore and docstore
vectorstore.save("path/to/vectorstore")
docstore.save("path/to/docstore")

# Later, reinitialize the retriever
vectorstore = FAISS.load("path/to/vectorstore", embedding=BedrockEmbeddings(
    client=self.get_bedrock_runtime_client(),
    model_id=model_id,
    normalize=True
))
docstore = LocalFileStore.load("path/to/docstore")

retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=docstore,
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)

In this example:

• vectorstore.save("path/to/vectorstore") and docstore.save("path/to/docstore") are placeholders for the actual methods to save the state of the vectorstore and docstore.
• FAISS.load("path/to/vectorstore", embedding=BedrockEmbeddings(...)) and LocalFileStore.load("path/to/docstore") are placeholders for the actual methods to load the state of the vectorstore and docstore.

Make sure to replace these placeholders with the actual methods provided by the FAISS and LocalFileStore classes to save and load their states.

For more detailed usage and examples, you can refer to the LangChain documentation on the ParentDocumentRetriever ^[1][2][3].

To continue talking to Dosu, mention @dosu.