Using Hybrid Search with ElasticsearchStore

[isspek]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_elasticsearch import ElasticsearchStore
from langchain_elasticsearch import DenseVectorStrategy
from langchain.docstore.document import Document
from langchain.schema import Document
from langchain_huggingface import HuggingFaceEmbeddings
import requests
import os
import logging

logging.basicConfig(level=logging.DEBUG)

def clear_index(es_url, index_name):
    requests.delete(f'{es_url}/{index_name}')

def main():
    es_url = 'http://localhost:9200'
    index_name = 'toy_index'

    clear_index(es_url, index_name)

    documents = [
        Document(
            page_content="A bunch of scientists bring back dinosaurs and mayhem breaks loose",
            metadata={"year": 1993, "rating": 7.7, "genre": "science fiction"},
        ),
        Document(
            page_content="Leo DiCaprio gets lost in a dream within a dream within a dream within a ...",
            metadata={"year": 2010, "director": "Christopher Nolan", "rating": 8.2},
        ),
        Document(
            page_content="A psychologist / detective gets lost in a series of dreams within dreams within dreams and Inception reused the idea",
            metadata={"year": 2006, "director": "Satoshi Kon", "rating": 8.6},
        ),
        Document(
            page_content="A bunch of normal-sized women are supremely wholesome and some men pine after them",
            metadata={"year": 2019, "director": "Greta Gerwig", "rating": 8.3},
        ),
        Document(
            page_content="Toys come alive and have a blast doing so",
            metadata={"year": 1995, "genre": "animated"},
        ),
        Document(
            page_content="Three men walk into the Zone, three men walk out of the Zone",
            metadata={
                "year": 1979,
                "director": "Andrei Tarkovsky",
                "genre": "science fiction",
                "rating": 9.9,
            },
        ),
    ]
    embeddings = HuggingFaceEmbeddings(
        model_name='nomic-ai/nomic-embed-text-v1',
        model_kwargs={'trust_remote_code': 'True'},
        encode_kwargs={'normalize_embeddings': True}
    )

    es_store = ElasticsearchStore(
        es_url=es_url,
        index_name=index_name,
        embedding=embeddings,
        strategy=DenseVectorStrategy(
            hybrid=True,
        )
    )
    es_store.from_documents(documents=documents, embedding=embeddings, index_name=index_name,
                            es_url=es_url)
    results = es_store.similarity_search("What are some movies about dinosaurs", k=1)
    for doc in results:
        print(doc.page_content)

if __name__ == "__main__":
    main()

Description

I would like to use the hybrid DenseVectorStrategy from ElasticSearch. I share a minimal code for this purpose.
I checked the documentation and implement it accordingly, but I am getting elasticsearch.BadRequestError: BadRequestError(400, 'parsing_exception', 'Unknown key for a START_OBJECT in [retriever].') in the line where search is operated.

The search in the code can work if I set hybrid option as False. But I would like to use hybrid option.

Much appreciated if you help on this issue. I searched for similar discussions, but those are answered by a bot, and does not help me fixing the issue.
Many thanks!

System Info

Here is the info about the installed libraries:

langchain==0.3.23
langchain-chroma==0.2.2
langchain-community==0.0.38
langchain-core==0.3.52
langchain-elasticsearch==0.3.2
langchain-huggingface==0.1.2
langchain-ollama==0.2.3
langchain-openai==0.3.9
langchain-text-splitters==0.3.8

The python version is Python 3.9.12.

Additionally the is the info about the docker that I used for the elastic search:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.1
    environment:
      - xpack.ml.enabled=true
      - xpack.security.enabled=false
      - discovery.type=single-node
      - ES_JAVA_OPTS=-Xmx8g
    ports:
      - "9200:9200"
    restart: "always"
    env_file:
      - ./.env
    networks:
      - elastic
    volumes:
      - ../volumes/elasticsearch:/usr/share/elasticsearch/data
    user: 1000:1000

[tuliochiodi-stellantis]

I was getting the same behavior using elasticsearch:8.12.1.
Using elasticsearch:8.18.2 solved, infortunetely I dont have commercial license so I had to use DenseVectorStrategy(hybrid=True, rrf=False).
Still need to evaluate the resulting retrieved documents to check if there is any improvements from regular non hybrid approach.