Dealing with duplicate documents to create an AzureSearch with a vector store

[pietz]

I want to create an Azure Cognitive Search application using a vector store for document embeddings plus additional meta data I can filter for. It's basically the second to last example on langchains Python Docs on Azure Search here.

There is one twist I have to deal with: A large number of meta data combinations will point to the same document. If I use langchain with AzureSearch providing an embedding function to create embeddings, I will create a very large number of duplicate embeddings because it will run through the same documents multiple times. Given that the OpenAI embedding API is not cheap for large databases, I really don't want that.

Any ideas how I can get around that?

[nadworny]

hi @pietz Have you found a solution for that?