[MEVD] Integrate with MEAI (Embedding and IEmbeddingGenerator)

[roji]

Background / Current status

MEVD is currently a separate package with no relation to Microsoft.Extensions.AI, which is where embedding-related types such as IEmbedding and IEmbeddingGenerator live. Because of this, MEVD doesn't currently expose APIs that make use of embedding types; the abstraction requires users to provide e.g. a ReadOnlyMemory<float>, requiring users to manually handle embedding generation outside MEVD. This makes MEVD a very low-level abstraction that's not easy to work with; a typical user of a vector database wants to set things up, and them simply run a vector search over e.g. some text, with embedding generation happening under the hood as an implementation detail. In other words, typical users should not need to deal directly with embeddings.

MEVD does contain IVectorizableTextSearch, which is an interface allowing doing a similarity search given a string; the string is vectorized automatically under the hood and the resulting embedding is sent. One reason for IVectorizableTextSearch is that (a) some database can take care of embedding generation themselves; connectors for such databases implement. The second reason (b) is to allow users to combine a vector database and a .NET embedding generator, exposing both via IVectorizableTextSearch implementation that runs the generator and passes the result to the database. The problem is, no such implementation for (b) currently exist - users have to provide an implementation, wire things up themselves in their application.

High-level proposal

First, in order for MEVD to provide a better, more high-level user experience, it must have access to IEmbedding/IEmbeddingGenerator. The proposal here is for MEVD to take a reference on MEAI in order to access these types; alternatives discussed where to move IEmbedding/IEmbeddingGenerator to 3rd place (M.E.Embeddings, or just System), and to merge MEVD and MEAI. I believe referencing MEAI from MEVD is non-controversial, so I won't go into the pros and cons of each option here (but can if needed).

DISCLAIMER: The below is my current view of the situation, I know @westey-m has a very different view (the point of this issue is to help hash things out)

• IVectorStoreRecordCollection should support a search method that accepts any .NET type, representing unvectorized user content (this is similar to IVectorizableTextSearch.VectorizedSearchAsync, but not text-only to support multi-modal vectorization). This to be the primary way of performing vector search via the abstraction (as opposed to the more low-level API accepting an embedding).
  • Naming TBD, but my preference would be something like SearchAsync, which the uneducated user would choose as the first/default option - we should guide users to this API.
• When setting up a IVectorStoreRecordCollection, the user should be able to optionally provide an IEmbeddingGenerator. For example, in DI configuration, an IEmbeddingGenerator service would be automatically picked up by the connectors' Add* methods. Then, when SearchAsync is called, that IEmbeddingGenerator would implicitly be used.
  • Note that an IVectorStoreRecordCollection can have multiple vector properties; as a result, the collection type is not generic over the vector embedding type, and can't be generic over the prevectorized input search type (e.g. string).
  • Since the SearchAsync method should accept any parameter type, and the input type to IEmbeddingGenerator can be anything, .NET typing does not enforce that the SearchAsync argument and the IEmbeddingGenerator input type match: a user can configure an IEmbeddingGenerator accepting a byte[], but call SearchAsync() passing a string; that's considered a user error, and this would cause a runtime exception.
  • Even when the .NET types align, IEmbeddingGenerators are implemented to support specific content types; if a user configures an IEmbeddingGenerator that expects a string base64 encoding of an image, but then passes text to SearchAsync, the resulting embeddings will be unusable (but no exception will be thrown - this is far worse).
  • A CLR type mismatch can also occur on the output side of the IEmbeddingGenerator as well: if an IEmbeddingGenerator is configured which outputs an embedding type that is unsupported by the vector database, an exception will be thrown. The same is true today when the user passes an embedding directly to MEVD: the user can provide whatever they want, and must know in advance what the database supports.
  • This all places the onus on the user the set things up correctly. It also means that swapping embedding generators isn't something that can be done lightly: if the new generator has a different input type, the user code will break. However, that will be the case regardless if the CLR type stays the same, but the content changes (different image type, or image vs. sound).
  • If no IEmbeddingGenerator was configured, calling SearchAsync will throw, and only the low-level API accepting embeddings can be used.
• A collection can have multiple vector properties; one could represent text, another an image - this requires the ability to configure multiple embedding generators.
  • MEVD has VectorStoreRecordDefinition, which represents schema/modeling information on a collection. It specifically contains VectorStoreRecordVectorProperty for defining metadata on vector properties, like the similarity function and index type to be used.
  • We can allow the user to set an IEmbeddingGenerator on VectorStoreRecordVectorProperty, to define the the generator used when SearchAsync is invoked over that property.
  • If the generator on a property isn't set, we can fall back to the one configured globally on the collection. In that way, e.g. a DI-registered IEmbeddingGenerator can act as the default for all vector properties, unless overridden. This will likely correspond to the 99% usecases, where a single text embedding model is for all properties across all collections; but still allows for overriding on a property-by-property basis.
  • MEVD also allows users to provide metadata (similarity function, index type) via attributes on the .NET properties; it will not be possible to configure a property-specific embedding generator via this mechanism. Again, this is expected to be an advanced/rare scenario. We will also have other things which aren't configurable via the attributes (e.g. anything provider-specific, #10359).
• The above scheme would require some changes/additions on the IEmbeddingGenerator side; to be usable from SearchAsync, we'd need the generic input type parameter to be on the GenerateAsync method (not on the type as it is currently). We'd also need the call to return a non-generic IEmbedding, since SearchAsync isn't generic over the specific embedding type (the non-generic IEmbedding would then passed to the vector database).

Notes

• To the best of my understanding, the main objection to this scheme is that since there's no standard representation for e.g. an image, it's impossible to swap an IEmbeddingGenerator and be sure that things still work; one might require the image to be provided as a base64 string, the other as a byte[].
  • MEVD is already - very much by design - a leaky abstraction that does not allow e.g. swapping databases. For example, different databases support different key types (string, long, Guid...); similarly, different databases support different embeddings (float32, bfloat, binary...).
  • Even if we dictate that images are e.g. always provided as a base64 string, this makes no guarantees about e.g. the image format (PNG vs. JPG), or even that the base64 string represents an image as opposed to sound. The CLR type matching doesn't guarantee that the content is what's expected, and garbage output embeddings seem like a much worse outcome than a clear, fail-fast exception about a CLR type mismatch.
  • I'm assuming that some embedding generators might support multiple input formats (PNG, JPG), but that when they do, they probably require some sort of additional metadata (e.g. mime type) to tell them what the image format is. The way in which metadata is provided will be generator-dependent, and so probably cannot be part of the abstraction.
• As a fundamental and relatively low-level abstraction, I don't believe MEVD should be making distinctions content types; for example, it should not have APIs that deal specifically with text, images or sound, have a sort of enum identifying content as images or sound, or be aware of MIME types. If we go down this route, we'll end up having to update the abstraction as new content types become relevant for vectorization. IMHO we should keep MEVD content-agnostic.
• Similarly, I don't believe MEVD should dictate what .NET type is used to encode e.g. an image. There are many things out there that may need to get vectorized (image, video, sound... who knows what else), and the abstraction shouldn't be in the business of knowing them or determining what representation is best. In addition, an encoding that is standard today (e.g. base64 image encoding) may go out of fashion tomorrow.
  • This is also affects API usability: I wouldn't want to force users to wrap e.g. a byte[] image in some wrapper we impose.
  • I can also see IEmbeddingGenerators evolving in terms of compositionality; a PDFEmbeddingGenerator implement could accept a PDF as input, internally extracting text out of the PDF and passing it to a second, wrapped IEmbeddingGenerator. Configuring such an IEmbeddingGenerator would allow users to directly provide a PDF document to SearchAsync (in whatever representation is standard in .NET), and have them be transparently vectorized etc.

/cc @markwallace-microsoft @dmytrostruk @stephentoub @adamsitnik

[westey-m]

I've done some more research to help ground our discussion. Here are my findings.

Multimodal Embedding models/services

Most services now provide multimodal embedding models.
This can take one of two forms:

1. The model can take an image or text as input, and produce an embedding in the same space, such that the output of the two can be compared successfully.
2. The model can take an image and/or text as input and generate a single embedding from the one/two inputs, allowing comparisons between any combination of these.

The best supported image embedding models I have been able to find have turned out to all be multi-modal models, so there does seem to be a trend towards these where image embedding is concerned. That said, multi-modal models do also support creating embeddings with images only, allowing image only usage if needed.

Image embedding generation services comparison

Here is a comparison of some models I've found that support image embedding.

Service	Img formats supported	Transport formats supported	Requires mime type	Multi-modal	Docs
Cohere Image Embedding	gif, jpeg, png, webp	Data URL	Y (as part of data URL)	Y	Docs
Cohere Image Embedding on Azure	gif, jpeg, png, webp	Data URL	Y (as part of data URL)	Y	Docs
Amazon Titan Multimodal Embeddings G1	jpeg, png	Base64 encoded string	N	Y	ModelInfo, Usage
Google Multimodal Embeddings	bmp, gif, jpeg, png	Remote ImageUri / Base64 encoded string via different properties	N	Y	Docs
Azure Computer Vision Multimodal	bmp, gif, jpeg, png, tiff	Remote ImageUri / Image Stream	Y for image stream	Y	ImgeUri Docs, ImageStream docs
Aleph Alpha	jpeg, ? (not documented)	base64 encoded string	N	Y	Docs, Sample with Qdrant

OpenAI has a multi-modal embedding model called CLIP as well, but I have not been able to find it as a hosted service. It is available as a Python package. See here for a sample.

Some observations from the comparison and my research:

1. Services support different transport formats for images.
2. Models support multiple image formats, with jpeg and png most widely supported.
3. Some services require the image type to be specified alongside the image data.
4. Google Multimodal Embeddings supports video as well, with many video formats supported simulatenously (AVI, FLV, MKV, MOV, MP4, MPEG, MPG, WEBM, and WMV)
5. Some services (Google, Azure Computer Vision) require you to use different fields/payloads depending on the chosen transport format, e.g. gcsUri parameter vs bytesBase64Encoded parameter

Some thoughts on the proposal

Here are some high level thoughts on the proposal.

Aside: In order to progress the discussion I will also need to think through the MEVD metadata story, but I need some more time to consider that fully, especially taking into account the rise of mult-modal.

Abstraction of the transport format for image embedding

The different embedding services do vary with regards to the transport format used for images. However these transport formats are bi-directionally convertable and converting is not complex.

Allowing users to provide images to the abstraction in a way that will work with any of these transport formats would increase the value of the abstraction significantly. Without this functionality, users would only be able to switch between different implementations without code changes, if those implementations support the same transport format.

When comparing with Vectors in VectorData, we do abstract away the different transport formats required by different Databases. E.g. while publicly we expose dense vectors as ReadOnlyMemory<T> this is converted to various formats by each connector, e.g. byte arrays in Redis.

Similar to image formats for image embedding, not all databases support all vector compression levels. However, there is a common, base level of support, that satisfies most customers, in the form of ReadOnlyMemory<float>. This allows the majority of customers to switch between databases without needing to make any code changes. Abstracting away the transport formats for images will allow the majority of customers, who will likely be using jpeg and png, to switch between embedding providers without code changes too.

Microsoft.Extensions.AI already support transport format abstraction for images passed to Chat Completion via the DataContent class.

I suggest that we follow a similar pattern for embedding generation.

Multi-modal support

We need to consider multi-modal embedding support as part of our embedding generation solution for images, considering the dominance of multi-modal models in the image embedding space. This means that our interfaces need to allow a user to indicate to us which parameter is an image and which is text, since we need this information when invoking the model.

Here is an example of the payload provided to the Amazon Titan G1 model. Note that both input_text and input_image are strings in this case.

    body = json.dumps({
        "inputText": input_text,
        "inputImage": input_image,
        "embeddingConfig": {
            "outputEmbeddingLength": output_embedding_length
        }
    })

Here is an example of the Google Multimodal Embeddings payload. As above, both text and image are strings.

{
  "instances": [
    {
      "text": "TEXT",
      "image": {
        "bytesBase64Encoded": "B64_ENCODED_IMG"
      }
    }
  ]
}

To support this type of service, we will either require a specialized interface+method for multi-modal embedding, or we need to have a well known type that can be passed to the generic IEmbeddingGenerator interface that allows the text and image to be provided separately.

We will of course also need to consider the addition of video and sound and potentially more in future.

[roji]

• @westey-m I agree with you that IEmbeddingGenerators which represent the actual services (which support non-text) should accept DataContent (or similar).
  • .NET simply lacks a (cross-platform) image abstraction, and in any case we need to support arbitrary other types (sound, video...); so a simple binary blob + MIME type (like DataContent) does seem like the way to go, like you suggested above.
  • A standard type like DataContent allow for easier swapping of embedding generator, as you suggested above.
  • It would also allow the generator to validate that the correct MIME type was provided.
• At the same time, I want to explore allowing intermediate (wrapping) IEmbeddingGenerator implementation, which would accept totally arbitrary inputs and convert them to DataContent. This isn't strictly-speaking mandatory, but I think it's a good usability improvement over only allowing DataContent/TextContent.
  • This allows various nice compositional scenarios:
    • An IEmbeddingGenerator implementation that accepts an ImageSharp or SkiaSharp representation of an image, converts it to a DataContent and passes that to the wrapped generator.
    • It could even go further and convert between image types based on what the wrapped generator actually supports (though that would require exposing supported MIME types on EmbeddingGeneratorMetadata).
    • Someone could also write an IEmbeddingGenerator that accepts some arbitrary business POCO, concatenates multiple fields from that and then generates an embedding from that.
    • In any case, after everything is set up, the user gets IVectorStoreRecordCollection that can search given whatever arbitrary .NET type makes sense in their application. That seems like a really nice dev experience.
  • This would mean that IEmbeddingGenerator doesn't constrain its TInput in any way, just like today. In effect, DataContent would be the recommended/de-facto standard input type for non-intermediate implementations (which represent actual services), but this wouldn't be enforced by the type system.
  • This also means that in MEVD, VectorStoreRecordCollection.SearchAsync should not be constrained to e.g. DataContent, but should allow any type.
    • That would mean having an unconstrained SearchAsync<T>(T input) on IVectorStoreRecordCollection.
    • The raw, embedding-based overload would need to be named differently (e.g. SearchEmbeddingAsync), since a SearchAsync(Embedding input) alongside the generic SearchAsync would only resolve for the Embedding type and not for subtypes.
    • I think that's OK, we want to generally guide people away from the embedding-based APIs and towards the ones performing vectorization internally.
  • Note that not constraining the input to e.g. DataContent would also allow text-only generators to accept a string directly, which seems desirable for that very common scenario.
• The above proposal brings us to the following situation:
  • Text-only generators accept string
    • We could in theory to take also support TextContent here for consistency/completeness, but I don't think it's necessary. An application dealing only with text just wants to use strings, and a switch to a multimodal generator would involve considerably more changes in any case.
  • Image-only generators accept DataContent
    • It may make sense to split DataContent to two types, one accepting the actual data (InlineDataContent?) and the other accepting a URL (UrlDataContent?). Currently, DataContent is a discriminated union of both, and splitting it would allow us to make a strong type distinction (e.g. CohereEmbeddingGeneration would accept only UrlDataContent as its input).
    • When passing in e.g. image data, in principle users should provide just provide DataContent wrapping byte[], and the implementation should take care of base64-encoding (or whatever) as an internal implementation detail. But if we think that users are likely to already have base64-encoded data from some other source, we can add that as an option to DataContent to avoid forcing the user to decode to byte[], only to re-encode back to base64 internally.
  • Multi-modal generators accept some base type that covers both DataContent and TextContent
    • As you pointed out, DataContent and TextContent currently extend AIContent, which also covers other LLM-specific types which make no sense in the MEVD context (e.g. FunctionCallContent).
    • We could introduce an intermediate abstract type as the base type for these two types, but I'm not sure it's worth it - it may be good enough to have multimodal generators accept AIContent, and throw if anything is given other than DataContent/TextContent. After all, even with an intermediate type, in theory someone could still create some new type extending it, and that would through at runtime. It feels like passing a FunctionCallContent to an IEmbeddingGenerator would at most be a silly mistake that programmers do once and then never again, so I'm not sure it's worth complicating the type hierarchy for that case.
  • Intermediate/wrapping generators accept anything.
• Note that validation remains a concern of the generator implementation - MEVD doesn't do any validation.
  • I think that's a good thing, since people using the generator directly without MEVD should get validation too.
  • MEVD also doesn't need to do any sort of automatic matching based on MIME type or similar - the user configures.
• Aside from all that, I think we previous comment notes on MEVD support remain relevant:
  • Users should be able to - but not required to - configure a generator for the collection; that gets used by default for all searches on that property.
  • For the rare cases where the collection has multiple properties with different generators, users should be able to configure a generator for each property.
  • If the non-embedding search is invoked without having configured a generator, an exception is thrown.
  • We need to integrate nicely and pick up an IEmbeddingGenerator configured in DI, etc.

Possible changes in MEAI for the above

• Introduce non-generic IEmbeddingGenerator
  * Make the TEmbedding type parameter of IEmbeddingGenerator<TInput, TEmbedding> covariant, so we can cast non-generic IEmbeddingGenerator down to IEmbeddingGenerator<TInput, Embedding> (note the concrete, non-generic Embedding here, which is the base type of all embeddings)
  • This unfortunately doesn't work, since IEmbeddingGenerator doesn't return a TEmbedding, but rather Task<ReturnedEmbeddings>. That’s OK, rather than trying to use covariance, the connector can simply cast down the non-generic IEmbeddingGenerator itself (the input type is already contravariant, the output type is a small closed list for the connector which can be tried). I’ll show this in our upcoming meeting.
• Possibly split DataContent into UrlDataContent and InlineDataContent. Not needed, no service actually accepts a URL for remote downloading, only URIs with embedded inline content.
• Possibly add an option for providing base64 content to InlineDataContent.
• Possibly introduce an intermediate base type above TextContent and DataContent (not yet convinced that's needed).

Approximate code sample to show the API

public virtual async Task<VectorSearchResults<TRecord>> SearchAsync<T>(
    T value,
    VectorSearchOptions<TRecord>? options = default,
    CancellationToken cancellationToken = default)
{
    var searchOptions = options ?? s_defaultVectorSearchOptions;
    var vectorProperty = this._propertyReader.GetVectorPropertyOrSingle(searchOptions.VectorPropertyName);
    var embeddingGenerator = vectorProperty.EmbeddingGenerator
                             ?? this._options.VectorStoreRecordDefinition?.EmbeddingGenerator
                             ?? this._store.Options.EmbeddingGenerator;

    switch (embeddingGenerator)
    {
        case IEmbeddingGenerator<T, Embedding<ReadOnlyMemory<float>>> generator:
            var embeddings = await generator.GenerateAsync([value], options: new() { Dimensions = vectorProperty.Dimensions }, cancellationToken)
                .ConfigureAwait(false);
            if (embeddings.Count != 1)
            {
                throw new InvalidOperationException($"The embedding generator returned {embeddings.Count} embeddings instead of the expected 1.");
            }

            return await this.SearchCoreAsync(embeddings.Single(), vectorProperty, operationName: "Search", options, cancellationToken)
                .ConfigureAwait(false);

        case null:
            // TODO: resx
            throw new InvalidOperationException("No embedding generator was configured");

        default:
            // TODO: resx
            throw new InvalidOperationException($"Expected embedding generator of type IEmbeddingGenerator<{typeof(T).Name}, Embedding<ReadOnlyMemory<float>>, but a generator of type {embeddingGenerator.GetType().Name} was configured");
    }
}

public virtual Task<VectorSearchResults<TRecord>> SearchEmbeddingAsync<TVector>(
    TVector vector,
    VectorSearchOptions<TRecord>? options = null,
    CancellationToken cancellationToken = default)
{
    ...
}