Calling LanceDB add_documents() repeatedly: each call overwrites previously written documents

[metadaddy]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_community.embeddings import GPT4AllEmbeddings
from langchain_community.vectorstores import LanceDB
from langchain_core.documents import Document

vectorstore = LanceDB(
    embedding=GPT4AllEmbeddings(model_name='all-MiniLM-L6-v2.gguf2.f16.gguf'),
)

doc1 = Document(page_content='Hello, world!')
vectorstore.add_documents([doc1])

doc2 = Document(page_content='Is there anybody out there?',)
vectorstore.add_documents([doc2])

row_count = vectorstore.get_table().count_rows()
print(f'Added 2 docs, but the row count is {row_count}')

# Output is 'Added 2 docs, but the row count is 1'

Description

Opening a discussion, because I'm not sure if the current LanceDB implementation is incorrect, or there are reasons for it working this way.

Current Situation

• I'm creating a LanceDB vector store and adding documents to it. I want to overwrite any existing LanceDB vector store at that location. The default mode in the LanceDB constructor is "overwrite", so I just do vectorstore = LanceDB(embedding=...).
• I then make multiple calls to add_documents(), but I find that each call overwrites the data written in previous calls.
• Examining the code reveals that I have to use mode="append" in the LanceDB constructor to have add_documents() actually add documents without deleting existing data.

This is quite unintuitive. With the default mode of "overwrite", I would expect any existing database to be overwritten, then successive add_documents() calls to actually add documents, rather than removing existing documents then adding the new ones.

To achieve my goal, I have to either:

• Manually delete the existing vector store, then create my LanceDB with mode="append", or
• Create my LanceDB with mode="overwrite", call add_documents() once, then create another LanceDB instance with mode="append" for subsequent add_documents() calls.

Further, if I later create another LanceDB with "overwrite" mode, then do vectorstore.get_table().count_rows(), there is data there. Existing data is not deleted until the first add_documents() call.

How I think it should work

• The default mode on creating a LanceDB object should be "append", rather than "overwrite", or perhaps there should be no default at all. It's better to unexpectedly append to existing data than unexpectedly replace it.
• Creating a LanceDB object with mode="overwrite" should delete the existing database.
• add_documents() should always add documents, rather than replacing the existing data.

System Info

System Information

OS: Darwin
OS Version: Darwin Kernel Version 23.6.0: Thu Sep 12 23:35:29 PDT 2024; root:xnu-10063.141.1.701.1~1/RELEASE_ARM64_T6000
Python Version: 3.12.3 (main, Sep 14 2024, 15:06:05) [Clang 15.0.0 (clang-1500.3.9.4)]

Package Information

langchain_core: 0.3.19
langchain: 0.3.7
langchain_community: 0.3.7
langsmith: 0.1.144
langchain_chroma: 0.1.4
langchain_openai: 0.2.9
langchain_text_splitters: 0.3.2
langchainhub: 0.1.21

Optional packages not installed

langgraph
langserve

Other Dependencies

aiohttp: 3.11.6
async-timeout: Installed. No version info available.
chromadb: 0.5.20
dataclasses-json: 0.6.7
fastapi: 0.115.5
httpx: 0.27.2
httpx-sse: 0.4.0
jsonpatch: 1.33
numpy: 1.26.4
openai: 1.55.0
orjson: 3.10.11
packaging: 24.2
pydantic: 2.9.2
pydantic-settings: 2.6.1
PyYAML: 6.0.2
requests: 2.32.3
requests-toolbelt: 1.0.0
SQLAlchemy: 2.0.35
tenacity: 8.5.0
tiktoken: 0.8.0
types-requests: 2.32.0.20241016
typing-extensions: 4.12.2

[RobFirth]

We are also encountering this issue and would like to see this clarified!

@metadaddy were you able to gain any understanding since you posted?

[metadaddy]

Hi @RobFirth - I just use the workaround I posted - if I want to overwrite the existing database, I delete the files with boto3, then create a LanceDB object with mode="append" so I can make multiple calls to add_documents().

It's unclear to me whether the issue is in LangChain's LanceDB integration or in LanceDB itself. I had hoped that this discussion might attract some of the LangChain maintainers to weigh in, but nothing so far. Maybe mentioning @ccurme might help...