How to filter metadata first and then perform similarity search by Chromadb?

[SetonLiang]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

def create_self_query_retriever(vectorstore: BaseRetriever) -> BaseRetriever:
    metadata_field_info = [
        AttributeInfo(
            name="faculty",
            description="The name of the faculty",
            type="string",
        ),
        AttributeInfo(name="source",description="The document this chunk is from.",type="string",),
    ]
    document_content_description = "Brief summary of faculty information"

    retriever = SelfQueryRetriever.from_llm(
        get_model(),
        vectorstore,
        document_content_description,
        metadata_field_info,
        verbose=True,
    )
    return retriever
result = vs_retriever.invoke("Where did Higgins Yang get his phd degree?")

Description

def create_self_query_retriever(vectorstore: BaseRetriever) -> BaseRetriever:
    metadata_field_info = [
        AttributeInfo(
            name="faculty",
            description="The name of the faculty",
            type="string",
        ),
        AttributeInfo(name="source",description="The document this chunk is from.",type="string",),
    ]
    document_content_description = "Brief summary of faculty information"

    retriever = SelfQueryRetriever.from_llm(
        get_model(),
        vectorstore,
        document_content_description,
        metadata_field_info,
        verbose=True,
    )
    return retriever

I want to first filter out documents in Chromadb where the metadata contains or matches the faculty name, and then perform a similarity search. I used SelfQueryRetriever, but retriever.invoke always shows filter=None. I want to know how to accurately filter custom attributes.
Alternatively, are there other methods to filter first and then perform similarity search?

System Info

pip freeze | grep langchain"
platform (windows / linux / mac)
python version

[feijoes]

@SetonLiang Maybe you're looking for search_kwargs={'filter': ....}}? I’m not sure if you’re looking to dynamically filter based on the metadata in the query or apply a static filter
docs

[SetonLiang]

The search_kwargs={'filter': ....}} will it first filter using the keyword and then retrieve documents based on similarity? Is it possible to avoid limiting the returned top_k?

[feijoes]

@SetonLiang

Sorry, but I'm not exactly sure how it works. According to the documentation, it says: 'filter: Filter by document metadata'. You could try checking the code manually to see what it does and create your own implementation if needed

[SetonLiang]

Okay, thanks a lot