Chromadb similarity search filter performance

[iPablo]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import chromadb


from fastapi import FastAPI, Request
from chromadb.config import Settings

from langchain_google_vertexai import VertexAIEmbeddings
from langchain_community.vectorstores import Chroma

app = FastAPI()

embedding_function = VertexAIEmbeddings(
        model_name="textembedding-gecko@003", requests_per_minute=150,
        project=f"{projectID}"
    )

app.client = chromadb.HttpClient(host="xx.xx.xx.xx",
                                ssl=False,
                                port=8000,
                                settings=(Settings(anonymized_telemetry=False)),)

app.vectordb = Chroma(
        client=app.client,
        collection_name=f"COLLECTION_NAME",
        embedding_function=embedding_function,
    )


@app.post("/test")
def index(request: Request):
    start_time = time.time()
    context_docs = app.vectordb.similarity_search_with_score(
                    query="Question here", k=100,
                    filter={"type": "QA"}
                )
    process_time = time.time() - start_time
    print(f"time: {process_time}")
    return f"{process_time} {context_docs}", 200, {"ContentType": "application/json"}

Description

I am making an api using FastAPI and langchain to answer a user Q&A. Doing performance tests I have realized that I have a bottleneck in the access to Chroma. At one point in the application I use the filter to only query documents marked as QA.

At that point the requests start to queue and drastically trigger the response time. Is there any way to make this filtering more efficient? Since querying all the documents is faster.

System Info

System Information

OS: Linux
OS Version: #1 SMP Fri Jan 19 08:53:17 UTC 2024
Python Version: 3.11.7 (main, Feb 1 2024, 04:47:46) [GCC 12.2.0]

Package Information

langchain_core: 0.1.19
langchain: 0.1.0
langchain_community: 0.0.11
langsmith: 0.0.87
langchain_google_vertexai: 0.0.1
langchain_openai: 0.0.5

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langgraph
langserve

[dosubot[bot]]

🤖

Hey @iPablo! Great to see you diving into another challenge. Hope you're doing well!

Based on the context provided, it seems like you're experiencing a performance issue when filtering documents marked as "QA" in your Chroma database. Here are a few strategies you might consider to improve the efficiency of your filter:

1. Batch Querying: The Chroma class in LangChain supports batch querying. This means that instead of querying one document at a time, you can query multiple documents at once. This can significantly reduce the time taken to query multiple documents.

2. Indexing: If you are querying documents based on certain metadata fields (like "QA"), it might be beneficial to index these fields. Indexing can significantly speed up the querying process. However, this might not be directly supported by Chroma and you might need to implement this at the database level.

3. Caching: If you find that you are querying the same documents frequently, you might want to consider implementing a caching mechanism. This way, you can store the results of a query in memory and retrieve them quickly if the same query is made again.

4. Parallel Processing: If you are dealing with a large number of queries, you might want to consider using parallel processing to speed up the querying process. This can be done using Python's multiprocessing module or other parallel processing libraries.

Regarding the similarity_search_with_score function in the Chroma class of LangChain, it handles filtering through the filter parameter. This parameter is an optional dictionary where the keys and values represent metadata fields and their respective values. The function uses this filter to narrow down the search results. Here is the relevant code snippet:

def similarity_search_with_score(
    self,
    query: str,
    k: int = DEFAULT_K,
    filter: Optional[Dict[str, str]] = None,
    where_document: Optional[Dict[str, str]] = None,
    **kwargs: Any,
) -> List[Tuple[Document, float]]:
    """Run similarity search with Chroma with distance.

    Args:
        query (str): Query text to search for.
        k (int): Number of results to return. Defaults to 4.
        filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None.

    Returns:
        List[Tuple[Document, float]]: List of documents most similar to
        the query text and cosine distance in float for each.
        Lower score represents more similarity.
    """
    if self._embedding_function is None:
        results = self.__query_collection(
            query_texts=[query],
            n_results=k,
            where=filter,
            where_document=where_document,
        )
    else:
        query_embedding = self._embedding_function.embed_query(query)
        results = self.__query_collection(
            query_embeddings=[query_embedding],
            n_results=k,
            where=filter,
            where_document=where_document,
        )
    return _results_to_docs_and_scores(results)

In this function, the filter parameter is passed to the __query_collection method, which is responsible for querying the Chroma database. If the filter parameter is provided, it will be used to filter the search results based on the metadata of the documents.

Please note that these are general approaches and their effectiveness can vary based on the specifics of your application and data. Always make sure to test any changes thoroughly to ensure they are improving performance as expected.

You can find more details about the similarity_search_with_score function in the Chroma class of LangChain in the source code.

Sources

• libs/langchain/langchain/vectorstores/chroma.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[issacchan26]

Hi, would you please provide an example of Indexing?

[GabrielPinheiroMatiucci]

@dosu-bot

Hi, could you show me an example of caching and one example of indexing?