How to apply metadata filter while retrieving docs from ElasticSearch

[amitguptadumka]

I have a index stored in Elastic search. Now, say for a query i want to retrieve top_k docs where i want to apply metadata filter and retrieval should take that into account.

I tried with below but it take complete URL value as match, how to do a partial match?
filters = MetadataFilters(filters=[ExactMatchFilter(key="URL", value="https://test/abc.com")])
retriever = VectorIndexRetriever(index=index, similarity_top_k=10,filters=filters)

For example my node contains metadata, metadata={'URL': 'https://test/abc.com'} and while retrieval i dont want to retrieve nodes which contains the word 'abc' in the URL part. How to do it, an example will be appreciated.

[dosubot[bot]]

🤖

Hello @amitguptadumka,

It's good to see you again and I hope you're doing well. I understand you're trying to apply a metadata filter for partial matching while retrieving documents from Elasticsearch using the LlamaIndex framework.

Currently, the LlamaIndex framework does not support partial matching for metadata filters in the ElasticsearchStore class. The _to_elasticsearch_filter function, which is used to convert standard filters to Elasticsearch filters, uses the "term" query for matching. The "term" query in Elasticsearch does exact matching and does not support partial matching.

Here is the relevant code snippet from the llama_index/vector_stores/elasticsearch.py file:

def _to_elasticsearch_filter(standard_filters: MetadataFilters) -> Dict[str, Any]:
    """Convert standard filters to Elasticsearch filter.

    Args:
        standard_filters: Standard Llama-index filters.

    Returns:
        Elasticsearch filter.
    """
    if len(standard_filters.filters) == 1:
        filter = standard_filters.filters[0]
        return {
            "term": {
                f"metadata.{filter.key}.keyword": {
                    "value": filter.value,
                }
            }
        }
    else:
        operands = []
        for filter in standard_filters.filters:
            operands.append(
                {
                    "term": {
                        f"metadata.{filter.key}.keyword": {
                            "value": filter.value,
                        }
                    }
                }
            )
        return {"bool": {"must": operands}}

In this function, the "term" query is used for each filter in the standard_filters.filters list. The "term" query does exact matching on the filter.value for the filter.key in the metadata.

Unfortunately, I wasn't able to find an alternative method for partial matching in metadata filters within the LlamaIndex repository. You might need to implement this functionality yourself by modifying the _to_elasticsearch_filter function to use a different Elasticsearch query that supports partial matching, such as the "match" or "wildcard" queries.

I hope this helps. If you have any other questions or need further clarification, feel free to ask.

Best regards.

Sources

• llama_index/vector_stores/elasticsearch.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.