Valid Azure Index structure for vector_store

[aayushrajj]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

vector_store: AzureSearch = AzureSearch(
    azure_search_endpoint=vector_store_address,
    azure_search_key=vector_store_password,
    index_name=index_name,
    embedding_function=embeddings.embed_query,
)

Description

Is there a defined field name and structrue that is accepted by vector_store langhcain while using Azure AI Index.
I am trying to upload chunks and thier respective embeddings on azure and retrieving context back.
I am getting while using this strcuture:

{
   "id":"---",
   "title":"----",
   "chunk":"----",
   "chunkVector":"---"
}

System Info

Windows
langchain=0.1.20
langchain-community=0.0.38
langchain-core=0.1.52
langchain-experimental=0.0.58
langchain-openai=0.1.6
langchain-text-splitters=0.0.2
azure-identity=1.17.1
azure-search-documents=11.4.0

[dosubot[bot]]

Hey, @aayushrajj! 👋 I'm here to help you with your bug, answer your questions, and maybe even inspire you to become a contributor!

Yes, there are defined field names and structures accepted by vector_store in LangChain while using Azure AI Index. The fields are:

• FIELDS_ID: Default is "id"
• FIELDS_CONTENT: Default is "content"
• FIELDS_CONTENT_VECTOR: Default is "content_vector"
• FIELDS_METADATA: Default is "metadata"

These fields can be overridden using environment variables AZURESEARCH_FIELDS_ID, AZURESEARCH_FIELDS_CONTENT, AZURESEARCH_FIELDS_CONTENT_VECTOR, and AZURESEARCH_FIELDS_TAG respectively.

Here is the relevant code snippet:

# Allow overriding field names for Azure Search
FIELDS_ID = get_from_env(
    key="AZURESEARCH_FIELDS_ID", env_key="AZURESEARCH_FIELDS_ID", default="id"
)
FIELDS_CONTENT = get_from_env(
    key="AZURESEARCH_FIELDS_CONTENT",
    env_key="AZURESEARCH_FIELDS_CONTENT",
    default="content",
)
FIELDS_CONTENT_VECTOR = get_from_env(
    key="AZURESEARCH_FIELDS_CONTENT_VECTOR",
    env_key="AZURESEARCH_FIELDS_CONTENT_VECTOR",
    default="content_vector",
)
FIELDS_METADATA = get_from_env(
    key="AZURESEARCH_FIELDS_TAG", env_key="AZURESEARCH_FIELDS_TAG", default="metadata"
)

The default fields structure is defined as follows:

default_fields = [
    SimpleField(
        name=FIELDS_ID,
        type=SearchFieldDataType.String,
        key=True,
        filterable=True,
    ),
    SearchableField(
        name=FIELDS_CONTENT,
        type=SearchFieldDataType.String,
    ),
    SearchField(
        name=FIELDS_CONTENT_VECTOR,
        type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
        searchable=True,
        vector_search_dimensions=vector_search_dimensions
        or len(self.embed_query("Text")),
        vector_search_profile_name="myHnswProfile",
    ),
    SearchableField(
        name=FIELDS_METADATA,
        type=SearchFieldDataType.String,
    ),
]

You should adjust your structure to match these field names and types ^[1].

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

Yes, search_kwargs is a valid parameter for AzureAISearchRetriever.

To apply a filter to the title field before retrieval, you can set the filter attribute directly in the AzureAISearchRetriever instance. Here is an example of how you can do this:

retriever = AzureAISearchRetriever(
    service_name="your_service_name",
    index_name="your_index_name",
    api_key="your_api_key",
    filter="title eq 'desired_title'"
)

This will filter the documents by the title field before performing the retrieval ^[1][2].

To continue talking to Dosu, mention @dosu.

[aayushrajj]

@dosu
no, it is still giving the same error.

ValidationError: 1 validation error for AzureAISearchRetriever
filter
  extra fields not permitted (type=value_error.extra)

[dosubot[bot]]

To correctly apply a filter to the title field when using AzureAISearchRetriever, you need to set the filter attribute within the search_kwargs parameter. The filter attribute should be part of the search_kwargs dictionary, which is passed to the AzureAISearchRetriever constructor.

Here is the correct way to set up the AzureAISearchRetriever with a filter:

from langchain_community.retrievers.azure_ai_search import AzureAISearchRetriever

# Initialize the retriever with necessary parameters
retriever = AzureAISearchRetriever(
    service_name="your_service_name",
    index_name="your_index_name",
    api_key="your_api_key",
    search_kwargs={
        "filter": "title eq 'desired_title'"
    }
)

# Use the retriever to get relevant documents
query = "your_search_query"
documents = retriever._get_relevant_documents(query=query, run_manager=None)

# Print the retrieved documents
for doc in documents:
    print(doc.page_content)

In this example, the filter attribute is set to "title eq 'desired_title'", which is an OData filter expression that filters documents where the title field is equal to 'desired_title'. This filter will be included in the search query and applied to the search results ^[1][2].

To continue talking to Dosu, mention @dosu.

[aayushrajj]

@dosu
Yes, I tried this but giving the error. It seems that there are only three key parameters for AzureAISearchRetriever.

# Initialize the retriever with necessary parameters
retriever = AzureAISearchRetriever(
    service_name="your_service_name",
    index_name="your_index_name",
    api_key="your_api_key",
    search_kwargs={
        "filter": "title eq 'desired_title'"
    }
)

[dosubot[bot]]

To correctly apply a filter to the title field before retrieval using AzureAISearchRetriever, you need to set the filter attribute directly, not within search_kwargs. Here is the correct way to initialize the AzureAISearchRetriever with a filter:

import os
from langchain_community.retrievers import AzureAISearchRetriever

# Set environment variables for Azure AI Search
os.environ["AZURE_AI_SEARCH_SERVICE_NAME"] = "your_service_name"
os.environ["AZURE_AI_SEARCH_INDEX_NAME"] = "your_index_name"
os.environ["AZURE_AI_SEARCH_API_KEY"] = "your_api_key"

# Initialize the retriever with necessary parameters
retriever = AzureAISearchRetriever(
    content_key="content",
    top_k=1,
    index_name=os.getenv("AZURE_AI_SEARCH_INDEX_NAME"),
    filter="title eq 'desired_title'"
)

# Use the retriever to get relevant documents
query = "your_search_query"
documents = retriever.invoke(query)

# Print the retrieved documents
for doc in documents:
    print(doc.page_content)

In this example, the filter is set to "title eq 'desired_title'", which will filter the documents to only those where the title field matches 'desired_title' ^[1][2][3].

To continue talking to Dosu, mention @dosu.