Skip to content

.Net: Added implementation of Azure CosmosDB for MongoDB connector for new memory design #7557

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 97 commits into from
Aug 21, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
97 commits
Select commit Hold shift + click to select a range
d41a1fa
.Net: Add Updated Memory Connector ADR and IMemoryRecordService inter…
westey-m Jun 5, 2024
cc59b04
.Net: Add new Azure AI Search memory connector implementation (#6585)
westey-m Jun 13, 2024
bfa0967
.Net: Rename memory to vector. (#6701)
westey-m Jun 17, 2024
0740d88
.Net: Adding unit tests for property enumerator and excluding from co…
westey-m Jun 17, 2024
c43ef3e
.Net: Adding generic redis IVectorRecordStore implementation (#6741)
westey-m Jun 21, 2024
15c1e9b
.Net: Add qdrant vector record store implementation (#6904)
westey-m Jun 24, 2024
17d55e9
.Net: VectorStore: Add unit tests for mappers and adding dedicated op…
westey-m Jun 26, 2024
ef5bef5
.Net: Add unit tests for AzureAISearch and Redis Vector Record Stores…
westey-m Jun 27, 2024
251218c
.Net: VectorStore: Adding unit tests for the Qdrant vector record sto…
westey-m Jun 28, 2024
2bfb66a
.Net: VectorStore: Add strongly typed properties for vector store exc…
westey-m Jul 1, 2024
4d8dcfd
.Net: VectorStore: Add custom serialization options / naming support.…
westey-m Jul 2, 2024
210d188
.Net: VectorStore: Moving all vector store interfaces to data folder …
westey-m Jul 3, 2024
3776975
.Net: Adding volatile vector record store implementation. (#7109)
westey-m Jul 8, 2024
21c94a7
.Net: Switching all vector record store instances to be tied to a sin…
westey-m Jul 10, 2024
9759ec3
.Net: Improve support for enumerable types and remove complex object …
westey-m Jul 15, 2024
6ffc0c7
.Net: Rename VectorRecordStore to VetorStoreRecordCollection (#7261)
westey-m Jul 15, 2024
0ef9b16
.Net: Adding collectionName, exists and delete to collection interfac…
westey-m Jul 16, 2024
eb54ca2
.Net: Adding Azure AI Search CollectionCreate (#7287)
westey-m Jul 16, 2024
c279270
.Net: Add redis collection create support. (#7315)
westey-m Jul 17, 2024
b2d34ee
Adding qdrant collection create. (#7302)
westey-m Jul 17, 2024
16dd962
.Net: Add CreateCollection to Volatile vectorstore and abstraction (#…
westey-m Jul 17, 2024
b0b6ece
.Net: Add IVectorStore interface and Volatile implementation (#7332)
westey-m Jul 17, 2024
ba8edc8
.Net: Adding VectorStore for Azure AI Search. (#7335)
westey-m Jul 18, 2024
6969ec0
.Net: Adding VectorStore for Redis. (#7345)
westey-m Jul 18, 2024
8909d87
.Net: Adding a VectorStore implementation for Qdrant (#7354)
westey-m Jul 18, 2024
9f71a4d
.Net: Remove mapper type option and default redis prefixing to true. …
westey-m Jul 19, 2024
288c57d
.Net: Fix casting bug in VolatileVectorStore (#7385)
westey-m Jul 22, 2024
7cce740
.Net: Add basic KernelBuilder and ServiceCollection extensions for re…
westey-m Jul 22, 2024
bb91cc6
.Net: Add Data ingestion sample for VectorStore (#7388)
westey-m Jul 23, 2024
a2dbb2f
.Net: Add missing experimental attributes and fix typos. (#7411)
westey-m Jul 24, 2024
1ed0970
.Net: dotnet Pinecone connector improvements: move to Pinecone.NET an…
maumar Jul 24, 2024
718dc5b
.Net: RedisJson rename, adr rename and AzureAISearch integration test…
westey-m Jul 24, 2024
11dce8f
.Net: Fix collection create field naming bug and add tests to cover s…
westey-m Jul 25, 2024
92220f2
.Net: Add support for redis hashsets. (#7445)
westey-m Jul 25, 2024
da26536
.Net: VectorStore: Add DI extensions for Pinecone and reorder params …
westey-m Jul 26, 2024
d8fd5e8
.Net: Removing embedding generation attributes and fixing cloning bug…
westey-m Jul 29, 2024
887fa0c
.Net: Add a few improvements to data ingestion sample. (#7524)
westey-m Jul 29, 2024
ef9bc1b
.Net: Fix bug where JsonSerializerOptions is not consistently used. (…
westey-m Jul 29, 2024
04e20ee
.Net: Add support for any key type to the VolatileVectorStore. (#7529)
westey-m Jul 29, 2024
e1b7fd2
Added initial implementation of collection and store classes
dmytrostruk Jul 29, 2024
b322798
Added integration test for vector store
dmytrostruk Jul 29, 2024
10d7ed9
Added special handling for _id property
dmytrostruk Jul 29, 2024
4dd7472
Implemented default mapping logic
dmytrostruk Jul 30, 2024
202e6d3
Fixes and more tests
dmytrostruk Jul 30, 2024
9703f8d
Merge branch 'feature-memory-service' into cosmos-db-upgrade
dmytrostruk Jul 30, 2024
948fd94
Fixes after merge
dmytrostruk Jul 30, 2024
e6f56ce
Fixed warnings
dmytrostruk Jul 30, 2024
8a591dd
.Net: Restricting property writability and renaming data model name p…
westey-m Jul 30, 2024
df0ea21
.Net: Rename DI extensions that depend on already registered dependen…
westey-m Jul 30, 2024
bb86fd7
Improved CollectionExists method
dmytrostruk Jul 31, 2024
74963d6
.Net: Removing reflection type checks where possible when using defin…
westey-m Jul 31, 2024
00e7c9a
Merge branch 'feature-memory-service' into cosmos-db-upgrade
dmytrostruk Jul 31, 2024
141e192
Fixes after merge
dmytrostruk Jul 31, 2024
fbb8814
Improved UpsertBatch method
dmytrostruk Jul 31, 2024
b913f35
Added logic for IncludeVectors property
dmytrostruk Jul 31, 2024
65cd240
.Net: [Feature branch] Small formatting improvements (#7588)
dmytrostruk Aug 1, 2024
2fec094
.Net: Make full text search configurable and improve support for tagg…
westey-m Aug 1, 2024
d7930b7
Merge branch 'feature-memory-service' into cosmos-db-upgrade
dmytrostruk Aug 1, 2024
8c35afd
Added unit tests for AzureCosmosDBMongoDBVectorStore
dmytrostruk Aug 1, 2024
03009da
Small fixes
dmytrostruk Aug 1, 2024
e628462
Added tests for collection class
dmytrostruk Aug 2, 2024
67bd925
Added unit tests for mapper
dmytrostruk Aug 2, 2024
c09bb40
Small fixes
dmytrostruk Aug 2, 2024
7e8f11c
Improved handling of index kind and distance functions
dmytrostruk Aug 2, 2024
99a3d64
Addressed PR feedback
dmytrostruk Aug 2, 2024
2327afc
Added logic for BSON serialization attributes
dmytrostruk Aug 5, 2024
9330ad3
Added methods for class registration in DI container
dmytrostruk Aug 5, 2024
99e834f
Merge from main
dmytrostruk Aug 6, 2024
f69feb0
Fix after merge
dmytrostruk Aug 6, 2024
eada3b6
Updated mapping logic. Added more integration tests
dmytrostruk Aug 6, 2024
e0f6abe
Addressed PR comment on index kind
dmytrostruk Aug 6, 2024
05783a3
Fixed path to vector property in index
dmytrostruk Aug 6, 2024
ddf6a89
Added unit tests for Get and Upsert methods with custom mapper
dmytrostruk Aug 6, 2024
8ef3a39
Merge branch 'main' into cosmos-db-upgrade
dmytrostruk Aug 6, 2024
e8f6875
Merge branch 'main' into cosmos-db-upgrade
dmytrostruk Aug 7, 2024
166bde4
Updated naming convention registration logic
dmytrostruk Aug 7, 2024
6b29874
Merge branch 'main' into cosmos-db-upgrade
dmytrostruk Aug 7, 2024
db547ee
Merge branch 'main' into cosmos-db-upgrade
dmytrostruk Aug 8, 2024
47cd158
Merge branch 'main' into cosmos-db-upgrade
dmytrostruk Aug 13, 2024
0e40dcb
Updated namespaces for DI extension methods
dmytrostruk Aug 13, 2024
98976db
Merge branch 'cosmos-db-upgrade' of https://github.com/dmytrostruk/se…
dmytrostruk Aug 13, 2024
de3e938
Small fix
dmytrostruk Aug 13, 2024
592b09f
Small improvement
dmytrostruk Aug 14, 2024
1afefa0
Merge branch 'main' into cosmos-db-upgrade
dmytrostruk Aug 14, 2024
9ae133e
Merge branch 'main' into cosmos-db-upgrade
dmytrostruk Aug 15, 2024
6ea6408
Merge branch 'main' into cosmos-db-upgrade
dmytrostruk Aug 16, 2024
c750b21
Updated XML documentation
dmytrostruk Aug 16, 2024
1bd61f8
Merge branch 'main' into cosmos-db-upgrade
dmytrostruk Aug 19, 2024
3760276
Merge branch 'main' into cosmos-db-upgrade
dmytrostruk Aug 19, 2024
9975434
Merge branch 'main' into cosmos-db-upgrade
dmytrostruk Aug 20, 2024
d7f03d9
Merge branch 'main' into cosmos-db-upgrade
dmytrostruk Aug 20, 2024
f675d45
Merge branch 'main' into cosmos-db-upgrade
dmytrostruk Aug 21, 2024
7b66bd8
Merge from main
dmytrostruk Aug 21, 2024
cca4ce1
Small fix
dmytrostruk Aug 21, 2024
5750737
Merge branch 'main' into cosmos-db-upgrade
dmytrostruk Aug 21, 2024
41c900f
Small fix in integration tests
dmytrostruk Aug 21, 2024
8fa98bf
Merge branch 'cosmos-db-upgrade' of https://github.com/dmytrostruk/se…
dmytrostruk Aug 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion dotnet/SK-dotnet.sln
Original file line number Diff line number Diff line change
Expand Up @@ -320,7 +320,9 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Connectors.Qdrant.UnitTests
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "StepwisePlannerMigration", "samples\Demos\StepwisePlannerMigration\StepwisePlannerMigration.csproj", "{38374C62-0263-4FE8-A18C-70FC8132912B}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "AIModelRouter", "samples\Demos\AIModelRouter\AIModelRouter.csproj", "{E06818E3-00A5-41AC-97ED-9491070CDEA1}"
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Connectors.AzureCosmosDBMongoDB.UnitTests", "src\Connectors\Connectors.AzureCosmosDBMongoDB.UnitTests\Connectors.AzureCosmosDBMongoDB.UnitTests.csproj", "{2918478E-BC86-4D53-9D01-9C318F80C14F}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "AIModelRouter", "samples\Demos\AIModelRouter\AIModelRouter.csproj", "{E06818E3-00A5-41AC-97ED-9491070CDEA1}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Connectors.AzureCosmosDBNoSQL.UnitTests", "src\Connectors\Connectors.AzureCosmosDBNoSQL.UnitTests\Connectors.AzureCosmosDBNoSQL.UnitTests.csproj", "{385A8FE5-87E2-4458-AE09-35E10BD2E67F}"
EndProject
Expand Down Expand Up @@ -799,6 +801,12 @@ Global
{38374C62-0263-4FE8-A18C-70FC8132912B}.Publish|Any CPU.Build.0 = Debug|Any CPU
{38374C62-0263-4FE8-A18C-70FC8132912B}.Release|Any CPU.ActiveCfg = Release|Any CPU
{38374C62-0263-4FE8-A18C-70FC8132912B}.Release|Any CPU.Build.0 = Release|Any CPU
{2918478E-BC86-4D53-9D01-9C318F80C14F}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{2918478E-BC86-4D53-9D01-9C318F80C14F}.Debug|Any CPU.Build.0 = Debug|Any CPU
{2918478E-BC86-4D53-9D01-9C318F80C14F}.Publish|Any CPU.ActiveCfg = Debug|Any CPU
{2918478E-BC86-4D53-9D01-9C318F80C14F}.Publish|Any CPU.Build.0 = Debug|Any CPU
{2918478E-BC86-4D53-9D01-9C318F80C14F}.Release|Any CPU.ActiveCfg = Release|Any CPU
{2918478E-BC86-4D53-9D01-9C318F80C14F}.Release|Any CPU.Build.0 = Release|Any CPU
{E06818E3-00A5-41AC-97ED-9491070CDEA1}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{E06818E3-00A5-41AC-97ED-9491070CDEA1}.Debug|Any CPU.Build.0 = Debug|Any CPU
{E06818E3-00A5-41AC-97ED-9491070CDEA1}.Publish|Any CPU.ActiveCfg = Debug|Any CPU
Expand Down Expand Up @@ -920,6 +928,7 @@ Global
{1D4667B9-9381-4E32-895F-123B94253EE8} = {0247C2C9-86C3-45BA-8873-28B0948EDC0C}
{E92AE954-8F3A-4A6F-A4F9-DC12017E5AAF} = {0247C2C9-86C3-45BA-8873-28B0948EDC0C}
{38374C62-0263-4FE8-A18C-70FC8132912B} = {5D4C0700-BBB5-418F-A7B2-F392B9A18263}
{2918478E-BC86-4D53-9D01-9C318F80C14F} = {0247C2C9-86C3-45BA-8873-28B0948EDC0C}
{E06818E3-00A5-41AC-97ED-9491070CDEA1} = {5D4C0700-BBB5-418F-A7B2-F392B9A18263}
{385A8FE5-87E2-4458-AE09-35E10BD2E67F} = {0247C2C9-86C3-45BA-8873-28B0948EDC0C}
EndGlobalSection
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Suppressing errors for Test projects under dotnet folder
[*.cs]
dotnet_diagnostic.CA2007.severity = none # Do not directly await a Task
dotnet_diagnostic.VSTHRD111.severity = none # Use .ConfigureAwait(bool) is hidden by default, set to none to prevent IDE from changing on autosave
dotnet_diagnostic.CS1591.severity = none # Missing XML comment for publicly visible type or member
dotnet_diagnostic.IDE1006.severity = warning # Naming rule violations
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
// Copyright (c) Microsoft. All rights reserved.

using System;
using System.Collections.Generic;
using Microsoft.SemanticKernel.Data;

namespace SemanticKernel.Connectors.AzureCosmosDBMongoDB.UnitTests;

public class AzureCosmosDBMongoDBHotelModel(string hotelId)
{
/// <summary>The key of the record.</summary>
[VectorStoreRecordKey]
public string HotelId { get; init; } = hotelId;

/// <summary>A string metadata field.</summary>
[VectorStoreRecordData]
public string? HotelName { get; set; }

/// <summary>An int metadata field.</summary>
[VectorStoreRecordData]
public int HotelCode { get; set; }

/// <summary>A float metadata field.</summary>
[VectorStoreRecordData]
public float? HotelRating { get; set; }

/// <summary>A bool metadata field.</summary>
[VectorStoreRecordData(StoragePropertyName = "parking_is_included")]
public bool ParkingIncluded { get; set; }

/// <summary>An array metadata field.</summary>
[VectorStoreRecordData]
public List<string> Tags { get; set; } = [];

/// <summary>A data field.</summary>
[VectorStoreRecordData]
public string? Description { get; set; }

/// <summary>A vector field.</summary>
[VectorStoreRecordVector(Dimensions: 4, IndexKind: IndexKind.IvfFlat, DistanceFunction: DistanceFunction.CosineDistance)]
public ReadOnlyMemory<float>? DescriptionEmbedding { get; set; }
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
// Copyright (c) Microsoft. All rights reserved.

using Microsoft.Extensions.DependencyInjection;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Connectors.AzureCosmosDBMongoDB;
using Microsoft.SemanticKernel.Data;
using MongoDB.Driver;
using Moq;
using Xunit;

namespace SemanticKernel.Connectors.AzureCosmosDBMongoDB.UnitTests;

/// <summary>
/// Unit tests for <see cref="AzureCosmosDBMongoDBKernelBuilderExtensions"/> class.
/// </summary>
public sealed class AzureCosmosDBMongoDBKernelBuilderExtensionsTests
{
private readonly IKernelBuilder _kernelBuilder = Kernel.CreateBuilder();

[Fact]
public void AddVectorStoreRegistersClass()
{
// Arrange
this._kernelBuilder.Services.AddSingleton<IMongoDatabase>(Mock.Of<IMongoDatabase>());

// Act
this._kernelBuilder.AddAzureCosmosDBMongoDBVectorStore();

var kernel = this._kernelBuilder.Build();
var vectorStore = kernel.Services.GetRequiredService<IVectorStore>();

// Assert
Assert.NotNull(vectorStore);
Assert.IsType<AzureCosmosDBMongoDBVectorStore>(vectorStore);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
// Copyright (c) Microsoft. All rights reserved.

using Microsoft.Extensions.DependencyInjection;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Connectors.AzureCosmosDBMongoDB;
using Microsoft.SemanticKernel.Data;
using MongoDB.Driver;
using Moq;
using Xunit;

namespace SemanticKernel.Connectors.AzureCosmosDBMongoDB.UnitTests;

/// <summary>
/// Unit tests for <see cref="AzureCosmosDBMongoDBServiceCollectionExtensions"/> class.
/// </summary>
public sealed class AzureCosmosDBMongoDBServiceCollectionExtensionsTests
{
private readonly IServiceCollection _serviceCollection = new ServiceCollection();

[Fact]
public void AddVectorStoreRegistersClass()
{
// Arrange
this._serviceCollection.AddSingleton<IMongoDatabase>(Mock.Of<IMongoDatabase>());

// Act
this._serviceCollection.AddAzureCosmosDBMongoDBVectorStore();

var serviceProvider = this._serviceCollection.BuildServiceProvider();
var vectorStore = serviceProvider.GetRequiredService<IVectorStore>();

// Assert
Assert.NotNull(vectorStore);
Assert.IsType<AzureCosmosDBMongoDBVectorStore>(vectorStore);
}
}
Loading
Loading