Dynamic MetaData Filtering for Vector Database Search

[innodyn1]

Greetings All,

Environment: Using Flowise 3.0.3 agentflow v2, self-hosted with Pinecone.

Issue: At this time, agentflow v2 is not capable of DYNAMIC metadata filtering.

I've set up multiple document stores in Flowise, each connected to a specific namespace within the same Pinecone database. The challenge is that each of these document stores contains different kinds of data, meaning their metadata filters vary significantly.

In my workflow, I'm using an LLM node I've called "Metadata Extractor." This LLM is given all the possible metadata filters and their values from every single document store. Its job is to parse keywords from the user's prompt and then match those keywords to the relevant metadata filters. I've instructed the LLM to ignore any metadata filters that don't have a value.

The output of this "Metadata Extractor" node is a JSON object. This JSON object contains only the key-value pairs for the metadata filters that are directly relevant to the user's prompt. I then assign this JSON output to a state variable called 'MetadataFilters'.

Unfortunately, I've hit a wall trying to pass this 'MetadataFilters' JSON object to Pinecone. The current API override configuration doesn't support this functionality. I've considered using a custom tool to call Pinecone, but that simply won't work with my current flow design.

My Suggestion: Dynamic Metadata Filtering
I strongly suggest adding an "Extract Metadata Retriever" node to AgentFlow v2. Here's how I envision it: within the configuration for document stores and knowledge (document stores) inside an agent node, provide an option to select a state variable (which would hold a JSON object of metadata key-value pairs). This would replace the need to manually use the current key-value editor for metadata filters. I believe this addition would address all my current and future use cases for dynamic metadata filtering.

Why This is Crucial
Without the ability to dynamically filter metadata, Flowise simply cannot be effectively used to create robust and complex RAG applications. This feature is essential for serious development.

Are there any immediate workarounds I might be overlooking for passing dynamic JSON metadata to Pinecone?