.Net: Adding VectorStore abstractions and 5 sample implementations #7595

westey-m · 2024-08-01T15:16:06Z

Motivation and Context

The current MemoryStore abstractions have limitations around consuming existing data stores with custom schemas.
This PR adds a new abstraction layer that supports custom schemas and data models, removes opinionated behaviors from connectors, and aims to support more of the features available in each database.

This set of abstractions do not yet contain any support for Searching, and this will be added to the IVectorStoreRecordCollection
interface soon.

See the included ADR document for more information.

Description

Adding IVectorStore and IVectorStoreRecordCollection interfaces, plus associated options classes, attributes and definition classes.
Adding an AzureAISearch implementation.
Adding a Qdrant implementation
Adding a Redis implementation with both JSON and Hashes support
Adding a Pinecone implementation
Adding a Volatile (In memory) implementation

Contribution Checklist

The code builds clean without any errors or warnings
The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
All unit tests pass, and I have added new tests where possible
I didn't break anyone 😄

…face with related classes. (#6364) Adding an ADR for evolving memory connectors. Related work item: #5887 ### Motivation and Context See included ADR for more motivation and context. ### Description See included ADR for more more info. ### Contribution Checklist  - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄

### Motivation and Context Adding a reference implementation using Azure AI Search for the new memory connector record interface. ### Description For more information see the included ADR document. Issue: #6519 ### Contribution Checklist  - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄

### Motivation and Context Renaming to the new vector store nomenclature as discussed. There are the main changes. All implementations follow the same pattern. IMemoryRecordService => IVectorRecordStore IMemoryRecordMapper => IVectorStoreRecordMapper MemoryServiceCommandExecutionException => VectorStoreOperationException MemoryDataModelMappingException => VectorStoreRecordMappingException MemoryRecordDataAttribute => VectorStoreRecordDataAttribute and same for other attributes MemoryRecordDataProperty => VectorStoreRecordDataProperty and same for other property definitions ### Description For more information see the included ADR document. Related Issue: #5887 ### Contribution Checklist  - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄

…de coverage since it's in InternalUtilities. (#6736) ### Motivation and Context The property enumerator helps to read property information from a model used with any of the vector stores. This property information is used to map from the storage model to the user data model. Adding unit tests for this property enumerator, and marking it with the exclude from code coverage attribute, since the file is located in the Internal Utilities folder and therefore is included in all projects, but it will not be unit tested in each, just in the one centralized location. ### Contribution Checklist  - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄

### Motivation and Context As part of the evolution of memory connectors, we need to support custom data models and remove opinionated behaviors, so adding a new record store implementation for redis. ### Description Adding an implementation for IVectorRecordStore for redis with support for: Custom mappers Generic data models Annotating data models via attributes or via definition objects. Also improving some styling in the AzureAISearch implementation. See #6525 ### Contribution Checklist  - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄

### Motivation and Context As part of the evolution of memory connectors, we need to support custom data models and remove opinionated behaviors, so adding a new record store implementation for qdrant. ### Description Adding an implementation for IVectorRecordStore for qdrant with support for: Custom mappers Generic data models Annotating data models via attributes or via definition objects. Also improving some styling in the AzureAISearch implementation. See #6525 ### Contribution Checklist  - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄

…tions for mapper. (#6948) ### Motivation and Context As part of adding new vector stores for redis and qdrant, these include mappers from data models to storage models and back, that require unit testing. The mapper interface also uses an options parameter which was reused from the IVectorRecordStore.GetAsync method, but really should have been a dedicated options object from the start. ### Description - Added unit tests for the redis and qdrant data model mappers. - Added a dedicated options model for the mapper. ### Contribution Checklist  - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄

#6968) ### Motivation and Context As part of updating the design of the memory connectors to allow custom schemas, adding unit tests for AzureAISearch and Redis Vector Record Stores. ### Description Add unit tests for AzureAISearch and Redis Vector Record Stores. ### Contribution Checklist  - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄

…re. (#6992) ### Motivation and Context As part of updating the design of the memory connectors to allow custom schemas, adding unit tests for Qdrant Vector Record Store. ### Description Add unit tests for Qdrant Vector Record Store. ### Contribution Checklist  - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄

…eption metadata. (#7002) ### Motivation and Context As part of the new vector store implementation, we need to add additional metadata to exceptions to share information about the operation that failed. ### Description Changing the way in which the metadata is stored to strongly typed properties. ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

…#7028) ### Motivation and Context As part of the new vector store implementation, we need to allow developers to provide their own names or json serialization options for storage properties. ### Description Adding support for this to AzureAISearch, Redis and Qdrant implementations in the following way: AzureAISearch: via JsonPropertyName attributes or JsonSerializerOptions. Redis: via JsonPropertyName attributes or JsonSerializerOptions. Qdrant: via StoragePropertyName on VectorRecord definitions. ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

…and namespace. (#7081) ### Motivation and Context We agreed to put all vector store core code in the Microsoft.SementicKernel.Data namespace, so moving all abstractions into this folder structure and namespace. ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

### Motivation and Context We need an in memory implementation of new vector store design, so allow simple usage of the pattern without needing to have an external database. ### Description This adds an In Memory / Volatile VectorRecordStore implementation and unit tests. ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

…gle collection (#7178) ### Motivation and Context As part of the new vector store design, we have decided to follow a pattern where each vector record store instance is tied to a single collection and we will have a separate factory interface for getting instances by collection name. ### Description - Removing optional collection names from all options classes. - Adding mandatory collection name to vector record store constructors. ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

…support from azure ai search. (#7210) ### Motivation and Context Supporting complex types is difficult in Azure AI Search for create via the abstraction, so for now, removing support for it, until someone requests it. This means doing type checking on data fields, so adding better support for enumerable types like Arrays, List<T>, IEnumerable<T>, etc. ### Description ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

### Motivation and Context After long discussion we decided to move to a model whereby we have a VectorStore that can provide VectorStoreRecordCollection instances. All record and collection specific operations will reside on VectorStoreRecordCollection and all cross collection operations will reside on VectorStore. ### Description This change renames VectorRecordStore to VectorStoreRecordCollection as agreed. The required collection operations will be moved to VectorStoreRecordCollection in a separate pr. ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

#7262) ### Motivation and Context As part of the memory connector redesign we have fixed on a design where we have a VectorStore that produces VectorStoreRecordCollection instances. These are tied to a collection and will expose single collection operations. ### Description This PR adds some of the single collection operations to the VectorStoreRecordCollection, namely: 1. CollectionExists 2. DeleteCollection As well as exposing the name of the collection. ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

### Motivation and Context As part of the memory connector redesign we have fixed on a design where we have a VectorStore that produces VectorStoreRecordCollection instances. These are tied to a collection and will expose single collection operations. ### Description This PR adds: - The ability to create Azure AI Search collections - Additional properties to the attributes and record definitions to support vector configuration and index creation. - Code to parse and map those new properties I'll update the IVectorStoreRecordCollection interface with the CreateCollection and CreateCollectionIfNotExists methods once I've added create to more implementations, to avoid having one very large pr. ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

### Motivation and Context As part of the memory connector redesign we have fixed on a design where we have a VectorStore that produces VectorStoreRecordCollection instances. These are tied to a collection and will expose single collection operations. ### Description This PR contains: - The ability to create Redis collections ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

### Motivation and Context As part of the memory connector redesign we have fixed on a design where we have a VectorStore that produces VectorStoreRecordCollection instances. These are tied to a collection and will expose single collection operations. ### Description This PR contains: - The ability to create Qdrant collections - Additional methods on mockable qdrant client to allow unit testing create - Had to move record property enumeration from the mapper to the collection class, since create requires the property information as well, specifically storage property names. - Removed the mapper options class, since the mapper is internal, and all the parameters are now required anyway. I'll update the IVectorStoreRecordCollection interface with the CreateCollection and CreateCollectionIfNotExists methods once I've added create to more implementations, to avoid having one very large pr. ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

…7318) ### Motivation and Context As part of the memory connector redesign we have fixed on a design where we have a VectorStore that produces VectorStoreRecordCollection instances. These are tied to a collection and will expose single collection operations. ### Description This PR contains: - The ability to create Volatile collections - Updating the abstraction to include the new methods. With this pr, all implementations now support create. ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

### Motivation and Context As part of the memory connector redesign we have fixed on a design where we have a VectorStore that produces VectorStoreRecordCollection instances. These are tied to a collection and will expose single collection operations. ### Description This PR adds: - The new IVectorStore interface that can serve collection instances and list collection names. - An in memory (Volatile) implementation of the new interface. ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

### Motivation and Context As part of the memory connector redesign we have fixed on a design where we have a VectorStore that produces VectorStoreRecordCollection instances. These are tied to a collection and will expose single collection operations. ### Description This pr adds - A IVectorStore implementation for AzureAISearch ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

### Motivation and Context As part of the memory connector redesign we have fixed on a design where we have a VectorStore that produces VectorStoreRecordCollection instances. These are tied to a collection and will expose single collection operations. ### Description This pr adds: - An IVectorStore implementation for Redis ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

### Motivation and Context As part of the memory connector redesign we have fixed on a design where we have a VectorStore that produces VectorStoreRecordCollection instances. These are tied to a collection and will expose single collection operations. ### Description This pr adds: - An IVectorStore implementation for Qdrant - Fix a bug in the mapping where using the json parser to create the caller object model needs json property names in order to correctly deserialize the json object. ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

…7356) ### Motivation and Context As part of the new VectorStore pattern, we support passing custom models to RecordCollection implementations. There was an enum to specify which one you want to use, default or custom, but we can just use the custom if provided and fall back to the built in version if not. Redis requires you to add a configurable prefix to record keys in order for the record to be indexed by an index. Whether you want prefixing to happen automatically is currently optional. Since records cannot be indexed without it, and configuring the setting is painful with the new vector store design, changing the default to true. ### Description - Removing mapper type enums - Switching redis prefix setting to true ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

### Motivation and Context The VolatileVectorStore uses a ConcurrentDictionary of ConcurrentDictionary instances to store records. Each collection can contain different types of records. It's not possible to cast the entire sub dictionary to the right generic type, so switching to casing on a per record basis. ### Description Fixing bug where casting at the dictionary level was causing a null reference exception. ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

…gistering VectorStores. (#7386) ### Motivation and Context As part of the memory connector redesign, we are adding new VectorStore classes. These should be easy to add to service collections both on the Kernel or outside. ### Description Adding KernelBuilder and ServiceCollection extension methods to register VectorStore instances for each of the VectorStore implementations, and adding unit tests for these as well. ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

### Motivation and Context Adding a sample that uses the new VectorStore implementations. Alternative suggestions for the best sample category for these are welcome. For now they are in the Memory folder. I have not updated the samples README.md yet, and will update, once we have a plan around where these should go. #7349 ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

### Motivation and Context Adding experimental attributes to artifacts that were missed before and fixing a few typos. ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

### Motivation and Context  As per dotnet/runtime#27939 (comment), small formatting improvements to use PascalCase for tuple return variables. In this case, the usage of such variables will be similar to C# property naming conventions (e.g. `properties.KeyProperty` instead of `properties.keyProperty`) ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

…ing. (#7593) ### Motivation and Context As part of the new VectorStore abstractions, it's possible to create collections with indexing. Users may want to explicitly configure which fields are full text searchable, since enabling this can add a lot of cost. See [#7560](#7560) ### Description - Adding option to make a property full text searchable - Adding implementations for AzureAISearch, Redis and Qdrant to support this option. - Updating Tagging support by enabling it where possible for enumerable string properties that are marked as filterable. ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

…7598) ### Description Adding two more examples for the VectorStore functionality: - A super simple example, showing just the most basic of functionality, that's easier to get started with. - An example which shows how to use a custom mapper for a case where your storage model and data model doesn't match or you need to optimize mapping. ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

...et/src/SemanticKernel.Abstractions/Data/RecordAttributes/VectorStoreRecordVectorAttribute.cs

dotnet/src/SemanticKernel.Abstractions/Data/RecordAttributes/VectorStoreRecordKeyAttribute.cs

dotnet/src/SemanticKernel.Abstractions/Data/RecordDefinition/DistanceFunction.cs

dotnet/src/SemanticKernel.Abstractions/Data/RecordDefinition/IndexKind.cs

dotnet/src/SemanticKernel.Abstractions/Data/RecordDefinition/VectorStoreRecordDataProperty.cs

dotnet/src/SemanticKernel.Abstractions/Data/RecordOptions/DeleteRecordOptions.cs

dotnet/src/SemanticKernel.Core/Data/KernelBuilderExtensions.cs

…#7618) ### Description A few improvements to the VectorStore xml docs based on suggestions from the team. ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

...t/src/Connectors/Connectors.Memory.AzureAISearch/AzureAISearchServiceCollectionExtensions.cs

dotnet/src/Connectors/Connectors.Memory.Pinecone/PineconeServiceCollectionExtensions.cs

dotnet/src/Connectors/Connectors.Memory.Qdrant/QdrantServiceCollectionExtensions.cs

dotnet/src/Connectors/Connectors.Memory.Redis/RedisServiceCollectionExtensions.cs

dotnet/src/SemanticKernel.Core/Data/ServiceCollectionExtensions.cs

### Description Fix service collection extensions return type typo. ### Contribution Checklist  - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄

### Motivation and Context Integration tests for redis are intermittently failing with connection errors, so increasing the connection timeout. ### Description Increase connection timeout for redis integration tests. ### Contribution Checklist  - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄

…icrosoft#7595) ### Motivation and Context The current MemoryStore abstractions have limitations around consuming existing data stores with custom schemas. This PR adds a new abstraction layer that supports custom schemas and data models, removes opinionated behaviors from connectors, and aims to support more of the features available in each database. This set of abstractions do not yet contain any support for Searching, and this will be added to the IVectorStoreRecordCollection interface soon. See the included ADR document for more information. ### Description - Adding IVectorStore and IVectorStoreRecordCollection interfaces, plus associated options classes, attributes and definition classes. - Adding an AzureAISearch implementation. - Adding a Qdrant implementation - Adding a Redis implementation with both JSON and Hashes support - Adding a Pinecone implementation - Adding a Volatile (In memory) implementation ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 --------- Co-authored-by: Maurycy Markowski <maumar@microsoft.com> Co-authored-by: Dmytro Struk <13853051+dmytrostruk@users.noreply.github.com>

westey-m added 30 commits July 25, 2024 15:44

dmytrostruk and others added 2 commits August 1, 2024 06:59

westey-m requested a review from a team as a code owner August 1, 2024 15:16

markwallace-microsoft added .NET Issue or Pull requests regarding .NET code kernel Issues or pull requests impacting the core kernel kernel.core documentation memory labels Aug 1, 2024

github-actions bot changed the title ~~Adding VectorStore abstractions and 5 sample implementations~~ .Net: Adding VectorStore abstractions and 5 sample implementations Aug 1, 2024

markwallace-microsoft approved these changes Aug 2, 2024

View reviewed changes

dmytrostruk approved these changes Aug 5, 2024

View reviewed changes

RogerBarreto approved these changes Aug 6, 2024

View reviewed changes

dmytrostruk approved these changes Aug 6, 2024

View reviewed changes

westey-m added this pull request to the merge queue Aug 6, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Aug 6, 2024

dmytrostruk approved these changes Aug 6, 2024

View reviewed changes

westey-m enabled auto-merge August 6, 2024 17:40

westey-m added this pull request to the merge queue Aug 6, 2024

westey-m removed this pull request from the merge queue due to a manual request Aug 6, 2024

westey-m added this pull request to the merge queue Aug 6, 2024

Merged via the queue into main with commit 1e982be Aug 6, 2024
37 of 39 checks passed

westey-m deleted the feature-memory-service branch August 6, 2024 17:59

westey-m restored the feature-memory-service branch August 6, 2024 18:17

crickman deleted the feature-memory-service branch January 6, 2025 17:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

.Net: Adding VectorStore abstractions and 5 sample implementations #7595

.Net: Adding VectorStore abstractions and 5 sample implementations #7595

Uh oh!

westey-m commented Aug 1, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

.Net: Adding VectorStore abstractions and 5 sample implementations #7595

.Net: Adding VectorStore abstractions and 5 sample implementations #7595

Uh oh!

Conversation

westey-m commented Aug 1, 2024

Motivation and Context

Description

Contribution Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!