Querying by metadata in vectorstore

[dokato]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_core.vectorstores import VectorStoreRetriever

meta_retr = VectorStoreRetriever(
    vectorstore=vectorstore,
    search_kwargs={"k": 1, "filter": {"page_id": {"$eq": f'{page_id}'}}}
)

Description

I'm building an agent that retrieves the page id from the content of the message. Now I want to query that id from my vector store, but I don't want to run similarity search on vectors, but instead just fetch a specific page_id based on the metadata in my store. Can I do that with langchain?

The example above return page that mentions my page_id in the content, and not the page of that particular page_id that I want.

System Info

langchain==0.1.4
langchain-community==0.0.16
langchain-core==0.1.16
langchain-openai==0.0.5
macos

[dosubot[bot]]

🤖

Hey @dokato! Great to see you back with another intriguing question. Hope you've been doing well since our last chat. 😊

Based on the information you provided, it seems like you're trying to retrieve a specific document from your vector store based on its page_id metadata. The VectorStoreRetriever class in LangChain currently doesn't support direct querying by metadata. It's primarily designed for similarity search on vectors.

However, you can achieve this by using the filter parameter in the similarity_search_with_score method of the PGVector class, as mentioned in the issue #13281. This method allows you to filter the results based on the metadata of the documents.

Here is an example of how you can use it:

from langchain_core.vectorstores import PGVector

# Assuming you have a connection string to your database
CONNECTION_STRING = "postgresql://user:password@localhost:5432/mydatabase"

# Create a PGVector instance
docsearch = PGVector.from_texts(
    texts=["foo", "bar", "baz"],
    collection_name="test_collection_filter",
    embedding=FakeEmbeddingsWithAdaDimension(),
    metadatas=[{"page_id": "1"}, {"page_id": "2"}, {"page_id": "3"}],
    connection_string=CONNECTION_STRING,
    pre_delete_collection=True,
)

# Search for a specific page_id
output = docsearch.similarity_search_with_score("foo", k=1, filter={"page_id": "1"})

In this example, the filter parameter is used to only retrieve documents that have the page_id "1". You can adjust the filter to match your needs. For example, you can filter by multiple metadata fields at once, or use more complex filter conditions.

Please note that the metadata fields you want to filter by need to be included in the metadatas parameter when creating the PGVector object. In the example above, the metadatas parameter includes a list of dictionaries, where each dictionary contains the metadata for one document. The keys in these dictionaries are the metadata fields, and the values are the corresponding metadata values for each document.

You can find more information about this in the test functions test_pgvector_with_metadatas(), test_pgvector_with_metadatas_with_scores(), test_pgvector_with_filter_match(), test_pgvector_with_filter_distant_match(), test_pgvector_with_filter_no_match(), test_pgvector_collection_with_metadata(), test_pgvector_with_filter_in_set(), test_pgvector_relevance_score(), test_pgvector_retriever_search_threshold(), and test_pgvector_retriever_search_threshold_custom_normalization_fn() in the libs/langchain/tests/integration_tests/vectorstores/test_pgvector.py file.

Please note that this approach will still perform a similarity search, but the results will be filtered based on the metadata. If you need to directly query by metadata without performing a similarity search, you might need to implement a custom solution outside of LangChain.

I hope this helps! If you have any further questions, feel free to ask.

Sources

• Utilize metadata during retrieval in ConversationalRetrieval and PGvector

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[mwwoo121]

I can't access :
https://github.com/langchain-ai/langchain/blob/master/libs/langchain/tests/integration_tests/vectorstores/test_pgvector.py

Would you point me to the tests for metadata search? Would like to check the functionality?