Can I delete all vectors belongs to a source document using Chroma vector DB?

[qingtian1771]

I want to delete all vectors belongs to a single source document in a chroma db, how can I do this?
I found following document at:
https://python.langchain.com/docs/integrations/vectorstores/chroma

# create simple ids
ids = [str(i) for i in range(1, len(docs) + 1)]

# add data
example_db = Chroma.from_documents(docs, embedding_function, ids=ids)
docs = example_db.similarity_search(query)
print(docs[0].metadata)

# update the metadata for a document
docs[0].metadata = {
    "source": "../../../state_of_the_union.txt",
    "new_value": "hello world",
}
example_db.update_document(ids[0], docs[0])
print(example_db._collection.get(ids=[ids[0]]))

# delete the last document
print("count before", example_db._collection.count())
example_db._collection.delete(ids=[ids[-1]])
print("count after", example_db._collection.count())

It seems we can delete vector using id, but how to find these ids according source (metadata)?
The reason for this function is there are many small documents I want to put into chroma, but I found some of them are wrong docs, so I have to delete them from chroma DB. And maybe I will find other wrong documents in the future.

Thanks!

[dosubot[bot]]

🤖

Hello,

To delete all vectors associated with a single source document in a Chroma vector database, you can indeed use the delete method provided by the Chroma class. However, you need to first identify the IDs of the vectors associated with the source document.

Here's how you can do it:

1. Iterate over all documents in the Chroma DB.
2. Check the metadata of each document.
3. If the source in the metadata matches the source document you want to delete, store the ID of that document.
4. Once you have all the IDs, you can call the delete method with these IDs.

Here's a sample code snippet:

# Assume you have a Chroma instance `chroma_instance` and the source document `source_doc`
ids_to_delete = []

for doc in chroma_instance:
    if doc.metadata.get('source') == source_doc:
        ids_to_delete.append(doc.id)

chroma_instance.delete(ids=ids_to_delete)

This will delete all vectors associated with the source document from the Chroma vector database.

Please note that this operation could be time-consuming if you have a large number of documents in your Chroma DB, as it involves iterating over all documents.

I hope this helps! Let me know if you have any other questions.

Sources

• libs/langchain/langchain/vectorstores/chroma.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[jaydee029]

@dosu-bot
how to do this for FAISS

[qingtian1771]

I think following code is not right, there is no "for doc in chroma_instance", am I right?

# Assume you have a Chroma instance `chroma_instance` and the source document `source_doc`
ids_to_delete = []

for doc in chroma_instance:
    if doc.metadata.get('source') == source_doc:
        ids_to_delete.append(doc.id)

chroma_instance.delete(ids=ids_to_delete)

[Fritz17236]

If using langchain.vectorstores.Chroma, calling .get() on your data base returns a dictionary with "ids", "metadatas" as keys. These are both lists with the same indices, e.g. metadatas[0] gives the metadata for document[0].

You can loop through the indices, scan the metadata and record the corresponding id:

coll = db.get()  # dict_keys(['ids', 'embeddings', 'documents', 'metadatas'])

ids_to_del = []

for idx in range(len(coll['ids'])):

    id = coll['ids'][idx]
    metadata = coll['metadatas'][idx]

    if metadata['source'] == "source_doc_to_remove":
        ids_to_del.append(id)

db._collection.delete(ids_to_del)

[dinakar17]

It's working

[qingtian1771]

Allow to specify ID when adding to the FAISS vectorstore.
#5190

How to delete or update a document within a FAISS index?
#2699

Just find some related issues, maybe we need a guide how to do this kind of things, maybe a blog.

[pushpendradahiya]

For your chroma DB, do the following:

ids = example_db.get(where = {'source': SOURCE_ID})['ids']
example_db.delete(ids = ids)

[Yarflam]

If you using langchain, do this:

# from langchain.vectorstores import Chroma
# my_db = Chroma(persist_directory=my_db_dest, embedding_function=embedding_function)

for collection in my_db._client.list_collections():
  ids = collection.get()['ids']
  print('REMOVE %s document(s) from %s collection' % (str(len(ids)), collection.name))
  if len(ids): collection.delete(ids)

[ghost]

saved my life, literally looking for this code for the past 2 hrs

[zhaziqwe]

save my life too, my boss force me to do this

[MixwelInc]

If you are using langchain and want to delete all vectors in collection, in my case it was easier to delete the collection:
from langchain.vectorstores import Chroma
vectorstore = Chroma( collection_name="full_documents_100", embedding_function=embedder)
vectorstore.delete_collection()

[ttww]

But after the delete_collection() you need to get the vectorstore again, because the langchain operation is not using the reset() of the ChromaDb collection....

[CHIRU98]

Do you have any ideas how to do the same with the qdrant db, in the qdrant there is no .get() support?

[javierhuertay]

What I did to solve the problem was:

# ids_to_delete = [id1,id2, ...]
chroma = Chroma(persist_directory=persist_directory, embedding_function=embedding_function)
chroma.delete(ids_to_delete) 
all_documents = chroma.get()
chroma.reset_collection() # Reset the collection to remove the deleted documents, otherwise they will be stored in the sqlite file
if all_documents["ids"]: #Check there are documents
      chroma.add_documents(all_documents) # Reinitialize only with the wanted documents

With the async functions work too