add support for OpenSearch vector store #508
Conversation
|
Hi @JM-Lab, AWS also provides OpenSearch and the java sdk provides AwsSdk2Transport. Wonder if should be part of this PR or a separate one. |
|
Hi @eddumelendez, Sure, use AwsSdk2Transport for OpenSearchClient. Check this link for details: https://opensearch.org/docs/latest/clients/java/#connecting-to-amazon-opensearch-service |
| <dependency> | ||
| <groupId>org.opensearch.client</groupId> | ||
| <artifactId>opensearch-java</artifactId> | ||
| <version>2.9.1</version> |
There was a problem hiding this comment.
Consider moving the version as a property in the parent POM
There was a problem hiding this comment.
Added a commit to move the version to a property in the parent POM.
| <version>1.0.0-SNAPSHOT</version> | ||
| <relativePath>../../pom.xml</relativePath> | ||
| </parent> | ||
| <artifactId>spring-ai-opensearch-store</artifactId> |
There was a problem hiding this comment.
And the spring-ai-opensearch-store dependency to the spring-bom
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.httpcomponents.client5</groupId> |
There was a problem hiding this comment.
Is this a test dependency only?
There was a problem hiding this comment.
It's a required dependency for OpenSearch client. See https://opensearch.org/docs/latest/clients/java/#installing-the-client-using-apache-httpclient-5-transport.
| } | ||
| } | ||
|
|
||
| public CreateIndexResponse createIndexMapping(String index, Map<String, Property> properties) { |
There was a problem hiding this comment.
is this supposed to be public?
There was a problem hiding this comment.
Yes, It's public to handle different sizes for each embedding model and allow for custom metadata mappings.
There was a problem hiding this comment.
Refer to the commit where I added the mappingJson parameter to OpenSearchVectorStore to handle different embedding sizes and allow custom metadata mappings.
|
Hi @tzolov, I plan to address your requests within the next 1-2 weeks. Would you prefer me to add commits to the current pull request or handle them in a separate pull request? Let me know what you think. |
|
- add concurrency to store.add(..) (bc embeddingClient is slow) - CassandraVectorStoreAutoConfiguration uses CassandraAutoConfiguration - driver profiles for production stability+performance, - small cleanups and naming fixes, - main doc tidy-up - astradb compatibility (protocol V4) – don't create embeddings again for documents that already have them similar to spring-projects#413
Facilitates the creation of multimodal message queries.
aligne logger name with the className
…pic3 at the same time.
Fixed java code.
…der and AwsRegionProvider
- Resolve an issue where a new index keeps getting created during application start up. - Solution is is to create an index only if an index on the embedding column does not exist. - Add missing index name.
…upport dynamic embedding request types.
…ms), and make index name unique (for when there are multiple vector indexes in the same keyspace) And change stream to for-loop when converting List<Double> to Float[] for performance
…pring-ai into opensearch-vector-store
…pring-ai into opensearch-vector-store
|
Hi @tzolov, Will reset tangled commits and soon add documentation for vectordbs with updates to catalog nav.adoc for a clean pull request. |
Add OpenSearchAiSearchFilterExpressionConverter
Add support for FilterExpression
Add support for OpenSearch vector store