.Net: OpenAI V2 & Assistants V2 - GA #7151

crickman · 2024-07-08T16:00:37Z

Accumulation of incremental changes that were reviewed separately.

This change fundamentally grapples with updating the Azure AI OpenAI SDK from V1 to V2.

This involves:

Including the OpenAI SDK dependency
Refactoring the OpenAI Connector
Introducing an Azure OpenAI Connector
Update the experimental Agents.OpenAI to support V2 assistant features (breaking change)

Empty Projects created for the following changes getting in place in small PRs. This pull request primarily includes changes to the .NET project files and the solution file. These changes introduce new projects and update package dependencies. The most significant changes are:

Empty Connectors.AzureOpenAI and Connectors.AzureOpenAI.UnitTests projects as a first step to start building AzureOpenAI conector based on new AzureOpenAI SDK. Co-authored-by: Dmytro Struk <13853051+dmytrostruk@users.noreply.github.com>

#6898) ## ClientCore + Foundation This PR is the first and starts the foundation structure and classes for the V2 OpenAI Connector. In this PR I also used the simpler `TextEmbeddings` service to wrap up a vertical slice of the Service + Client + UT + IT + Dependencies needed also to validate the proposed structure of folders and namespaces for internal and public components. ### ClientCore As part of this PR I'm also taking benefit of the `partial` keyword for `ClientCore` class dividing its implementation per Service. In the original V1 ClientCore the file was very big, and creating specific files PR service/modality will make it simpler and easier to maintain. ## What Changed This change includes a new update from previous `Azure.Core` Pipeline abstractions to the new `System.ClientModel` which is used by `OpenAI` package. Those include the update and addition of the below files: - AddHeaderRequestPolicy - Adapted from previous `AddHeaderRequestPolicy` - ClientResultExceptionExtensions - Adapted from previous `RequestExceptionExtensions` - OpenAIClientCore - Merged with ClientCore (No more need for a specialized Azure and OpenAI clients) - ClientCore (Updated internals just with necessary for Text Embeddings), merged `OpenAIClientCore` also into this one and made it not as `abstract` class. - OpenAITextEmbbedingGenerationService (Updated to use `ClientCore` directly instead of `OpenAIClientCore`. ## Whats New - [PipelineSynchronousPolicy - Azure.Core/src/Pipeline/HttpPipelineSynchronousPolicy.cs](https://github.com/Azure/azure-sdk-for-net/blob/8bd22837639d54acccc820e988747f8d28bbde4a/sdk/core/Azure.Core/src/Pipeline/HttpPipelineSynchronousPolicy.cs#L18) This file didn't exist and was necessary to add as it is a dependency for `AddHeaderRequestPolicy` - Mockups added for `System.ClientModel` pipeline testing - Unit Tests Covering - ClientCore - OpenAITextEmbeddingsGenerationService - AddHeadersRequestPolicy - PipelineSynchronousPolicy - ClientResultExceptionExtensions - Integration Tests - OpenAITextEmbeddingsGenerationService (Moved from V1) ## What was Removed - OpenAIClientCore - This class was merged in ClientCore - CustomHostPipelinePolicy - Removed as the new OpenAI SDK supports Non-Default OpenAI endpoints. ## Unit & Integration Test Differently from V1, this PR focus on individual UnitTest for the OpenAI connector only. With the target of above 80% code converage the Unit Tests targets Services + Clients + Extensions & Utilities The structure of folders and tested components on the UnitTests will follow the same structure defined in project under test.

…s.OpenAI to Connectors.AzureOpenAI (#6906) ### Motivation and Context As a first step in migrating AzureOpenAIConnector to Azure AI SDK v2, all code related to AzureOpenAIChatCompletionService, including unit tests, is copied from the Connectors.OpenAI project to the Connectors.AzureOpenAI project as-is, with only the structural modifications described below and no logical modifications. This is a preparatory step before refactoring the AzureOpenAIChatCompletionService to use Azure SDK v2. ### Description This PR does the following: 1. Copies the AzureOpenAIChatCompletionService class and all its dependencies to the Connectors.AzureOpenAI project as they are, with no code changes. 2. Copies all existing unit tests related to the AzureOpenAIChatCompletionService service and its dependencies to the Connectors.AzureOpenAI.UnitTests project. 3. Renames some files in the Connectors.AzureOpenAI project so that their names begin with AzureOpenAI instead of OpenAI. 4. Changes namespaces in the copied files from Microsoft.SemanticKernel.Connectors.OpenAI to Microsoft.SemanticKernel.Connectors.AzureOpenAI. Related to the "Move reusable code from existing Microsoft.SemanticKernel.Connectors.OpenAI project to the new project" task of the #6864 issue. ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

- Updated ImageToText service implementation using OpenAI SDK - Updated ImageToText service API's parameters order (modelId first) and added modelId as required (OpenAI supports both dall-e-2 and dall-e-3) - Added support for OpenAIClient breaking glass for Image to Text Service - Added support for custom/Non-default endpoint for Image to Text Service - Added missing Extensions (Service Collection + Kernel Builder) for Embeddings and Image to Text modalities - Added missing UnitTest for Embeddings - Added UT convering Image to Text. - Added integration tests for ImageTotext - Resolve Partially #6916

- Updating policies using OpenAI SDK approach (GenericPolicy) impl. - Updated Unit Tests - Moved policy impl to openai Utilities.

) ### Motivation and Context This PR is the next step in a series of follow-up PRs to migrate AzureOpenAIConnector to Azure AI SDK v2. It updates all code related to AzureOpenAI ChatCompletionService to use the Azure AI SDK v2. One of the goals of the PR is to update the code with a minimal number of changes to make the code review as easy as possible, so almost all methods keep their names as they were even though they might not be relevant anymore. This will be fixed in one of the follow-up PRs. ### Description This PR does the following: 1. Migrates AzureOpenAIChatCompletionService, ClientCore, and other model classes both use, to Azure AI SDK v2. 2. Updates ToolCallBehavior classes to return a list of functions and function choice. This change is required because the new SDK model requires both of those for the CompletionsOptions class creation and does not allow setting them after the class is already created, as it used to allow. 3. Adapts related unit tests to the API changes. ### Next steps 1. Add integration tests. 2. Rename internal/private methods that were intentionally left with old, irrelevant names to minimize the code review delta. ### Out of scope: * #6991 ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄

…tionService v2 (#7003) ### Motivation and Context This PR is the next step in a series of follow-up PRs to migrate the AzureOpenAIChatCompletion service to the Azure AI SDK v2. It adds extension methods for the service collection and kernel builder to create and register the AzureOpenAIChatCompletionService. Additionally, it includes integration tests for the service. ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 --------- Co-authored-by: Dmytro Struk <13853051+dmytrostruk@users.noreply.github.com>

…ureOpenAI project (#7022) ### Motivation and Context This PR prepares the AzureOpenAITextEmbeddingGenerationService for migration to the new Azure AI SDK v2. The AzureOpenAITextEmbeddingGenerationService is copied to the Connectors.AzureOpenAI project as is and excluded from compilation to simplify code review of the next PR, which will refactor it to use the new Azure SDK. The next PR will also add unit/integration tests, along with service collection and kernel builder extension methods. ### Description The AzureOpenAITextEmbeddingGenerationService class was copied as is to the Connectors.AzureOpenAI project. The class build action was set to none to exclude it temporarily from the compilation process until it gets refactored to the new SDK.

### Motivation and Context This PR is a follow-up to the one that migrated the AzureOpenAIChatCompletionService class to Azure AI SDK v2 but did not properly refactor it, leaving class, member, and variable names unchanged to minimize the number of changes and simplify the code review process. ### Description This PR does the following: **1. No functional changes - just deletion and renaming.** 2. Renames ClientCore class methods and method variables to reflect their actual purpose/functionality. 3. Renames class members of other classes related to and used by the AzureOpenAIChatCompletionService. 4. Renames AzureOpenAIPromptExecutionSettings class to AzureOpenAIChatCompletionExecutionSettings to indicate that it belongs to the chat completion service and not to any other one. 5. Removes the AzureOpenAIStreamingTextContent class used by the text generation service, which has been deprecated and is not supported by Azure AI SDK v2. 6. Improves the resiliency of AzureOpenAIChatCompletionService integration tests by using a base integration class with a preconfigured resilience handler to handle 429 responses.

…DK v2 (#7030) ### Motivation, Context and Description This PR migrates the AzureOpenAITextEmbeddingGenerationService to Azure AI SDK v2: 1. The AzureOpenAITextEmbeddingGenerationService class is updated to use the new AzureOpenAIClient from Azure AI SDK v2. 2. Service collection and kernel builder extension methods for registering the service are copied to the new Connectors.AzureOpenAI project. 3. Unit tests are added (copied and adapted) for the service and the extension methods. 4. Integration tests are added (copied and adapted) for the service.

…7048) ### Motivation, Context and Description This PR moves the AzureOpenAIChatCompletionService from the ChatCompletion folder to the Services folder to align the AzureOpenAI project structure with that of OpenAIV2. Additionally, the AzureOpenAIChatCompletionServiceTests unit tests were also moved to the appropriate folder for consistency.

### Motivation and Context - Audio to Text Services + UT + IT - Text to Audio Services + UT + IT - Added missing extension methods for Breaking Glass `OpenAIClient` - Moved ClientResultExtension + UT to utilities - Added and Moved Extensions + UT

### Motivation and Context The ClientCore class has absorbed functionality relevant to different Azure OpenAI-related services. As a first step to reduce its size, it makes sense to split it into small chunks, where each chunk would be related to a service that uses it. Later, we can move those chunks to the services themselves. ### Description This PR does the following: 1. Does not change any logic in any of the services. 2. Splits the `ClientCore` class into `ClientCore.ChatCompletion` and `ClientCore.Embeddings` files. 3. Refactors the `AzureOpenAIChatCompletionService` and `AzureOpenAITextEmbeddingGenerationService` to use the relevant `ClientCore` pieces. 4. Removes the `AzureOpenAIClientCore` class

### Motivation, Context and Description This PR fixes a small issue that would occur if the LLM started calling tools that are not functions. Additionally, it renames the `openAIClient` parameter of the `AzureOpenAIChatCompletionService` class constructor to `azureOpenAIClient` to keep the name consistent with its type. It also renames the `AzureOpenAIChatMessageContent.GetOpenAIFunctionToolCalls` method to `GetFunctionToolCalls` because the old one is not relevant anymore. The last two changes are breaking changes and it will be decided in the scope of the #7053 issue whether to keep them or roll them back.

### Motivation, Context and Description This PR removes the duplicate ClientResultExceptionExtensions extension class from the new AzureOpenAI project in favor of the existing extension class in the utils. It also includes the utils class in the SK solution, making it visible in the Visual Studio Explorer.

…nto separate classes. (#7078) ### Motivation, Context and Description This PR moves Azure-specific kernel builder extension methods from the `AzureOpenAIServiceCollectionExtensions` class to a newly introduced one - `AzureOpenAIKernelBuilderExtensions`, **as they are, with no functional changes** to follow the approach taken in SK - one file/class per type being extended.

…ct (#7077) ### Motivation, Context and Description This PR copies the `OpenAITextToImageService` class and related code, including unit tests, from the `Connectors.OpenAIV2` project to the `Connectors.AzureOpenAI` project. The copied classes have no functional changes; they have only been renamed to have the `Azure` prefix and placed into the corresponding `Microsoft.SemanticKernel.Connectors.AzureOpenAI` namespace. All the classes are temporarily excluded from the compilation process. This is done to simplify the code review of follow-up PR(s) that will include functional changes. A few small fixes, unrelated to the main purpose of the PR, were made to the XML documentation comments and logging in the `OpenAIAudioToTextService` and `OpenAITextToImageService` classes.

) ### Motivation, Context, and Description This PR migrates `AzureOpenAITextToImageService` to Azure.AI.OpenAI v2: 1. It updates the previously added `AzureOpenAITextToImageService` to use `AzureOpenAIClient`. 2. It replaces all constructors in the `AzureOpenAITextToImageService` with the relevant ones from the original `AzureOpenAITextToImageService`. 3. It adds the `serviceVersion` parameter to the `ClientCore.GetAzureOpenAIClientOptions` methods, allowing the specification of the service API version. 4. It updates XML documentation comments in a few classes to indicate their relevance to Azure OpenAI services. 5. It adds unit tests for the `AzureOpenAITextToImageService` class.

### Motivation and Context - Added FileService OpenAI V2 Implementation - Updated Extensions and UT accordingly - Updated ClientCore to be used with FileService and allow non-required `modelId`.

…project (#7099) ### Motivation, Context and Description This PR copies the existing `AzureOpenAITextToAudioService` class from the `Connectors.OpenAI` project and `ClientCore.TextToAudio` class from `Connectors.OpenAIV2` to the `Connectors.AzureOpenAI` project. The copied classes have no functional changes; they have only been renamed to include the Azure prefix and placed into the corresponding Microsoft.SemanticKernel.Connectors.AzureOpenAI namespace. All the classes are temporarily excluded from the compilation process. This is done to simplify the code review of follow-up PR(s) that will include functional changes.

…#7097) ### Motivation, Context, and Description This PR adds service collection and kernel builder extension methods that register the newly added `AzureOpenAITextToImageService`. The method signatures remain the same as the current extension methods, except for those that had an `OpenAIClient` parameter, whose name has been changed from `openAIClient` to `azureOpenAIClient` and whose type has been changed to `AzureOpenAIClient`. The breaking change is tracked in this issue: #7053. Additionally, this PR adds unit tests for the extension methods and integration tests for the service.

### Motivation and Context A few cosmetic improvements were identified while migrating {Azure}OpenAI services to Azure.AI.OpenAI SDK v2. This PR implements those improvements.

…e.AI.OpenAI SDK V2 (#7112) ### Motivation, Context, and Description This PR copies existing AzureOpenAIAudioToTextService-related classes to the new Connectors.AzureOpenAI project as they are, with only the namespaces changed. The classes are temporarily excluded from the compilation process and will be included again in a follow-up PR. This is done to simplify the review process of the next PR.

…#7102) ### Motivation and Context This PR migrates AzureOpenAITextToAudioService to Azure.AI.OpenAI SDK v2. ### Description 1. The new `AzureOpenAITextToAudioExecutionSettings` class is added to represent prompt execution settings for `AzureOpenAITextToAudioService`. Both `AzureOpenAITextToAudioService` classes that SK has today use the same prompt execution settings class - [OpenAITextToAudioExecutionSettings](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/Connectors/Connectors.OpenAI/TextToAudio/OpenAITextToAudioExecutionSettings.cs). This is a breaking change that is tracked in the issue - #7053, and it will be decided later whether to proceed with the change or roll it back. 2. The `ClientCore.TextToAudio.cs` class is refactored to use the new `AzureOpenAITextToAudioExecutionSettings` class. 3. The `ClientCore.TextToAudio.cs` class is refactored to decide which model id to use - the one from the prompt execution setting, the one supplied when registering the connector, or to use the deployment name if no model id is provided. This is done for backward compatibility with the existing `AzureOpenAITextToAudioService`. #7104 4. Service collection and kernel builder extension methods are added to register the service in the DI container. 5. Unit and integration tests are added as well.

Accumulation of incremental changes that were reviewed separately. This change fundamentally grapples with updating the Azure AI OpenAI SDK from V1 to V2. This involves: - Including the OpenAI SDK dependency - Refactoring the OpenAI Connector - Introducing an Azure OpenAI Connector - Update the experimental `Agents.OpenAI` to support V2 assistant features (breaking change) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Roger Barreto <19890735+RogerBarreto@users.noreply.github.com> Co-authored-by: SergeyMenshykh <sergemenshikh@gmail.com> Co-authored-by: SergeyMenshykh <68852919+SergeyMenshykh@users.noreply.github.com> Co-authored-by: Dmytro Struk <13853051+dmytrostruk@users.noreply.github.com> Co-authored-by: Roger Barreto <rbarreto@microsoft.com> Co-authored-by: Dr. Artificial曾小健 <875100501@qq.com> Co-authored-by: westey <164392973+westey-m@users.noreply.github.com> Co-authored-by: Mark Wallace <127216156+markwallace-microsoft@users.noreply.github.com> Co-authored-by: Tao Chen <taochen@microsoft.com> Co-authored-by: Evan Mattson <35585003+moonbox3@users.noreply.github.com> Co-authored-by: Maurycy Markowski <maumar@microsoft.com> Co-authored-by: gparmigiani <jkone27@users.noreply.github.com> Co-authored-by: Atiqur Rahman Foyshal <113086917+atiq-bs23@users.noreply.github.com> Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com> Co-authored-by: Andrew Desousa <33275002+andrewldesousa@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Marcelo Garcia <129542431+MarceloAGG@users.noreply.github.com> Co-authored-by: Marcelo Garcia 🛸 <marcgarc@microsoft.com> Co-authored-by: Eirik Tsarpalis <eirik.tsarpalis@gmail.com>

arynaq · 2024-09-04T12:57:57Z

Yai, looks promising, been looking forward to using agents (for cases where data sensitivity is not an issue, I wish MS gave an indepth insight into how persistence is handled for threads and agents so my data officer can get some sleep)

RogerBarreto and others added 30 commits June 20, 2024 19:06

feat: add empty AzureOpenAI and AzureOpenAI.UnitTests projects.

00f80bc

fix: temporarily disable package validation

5cd0a28

fix: publish configuration for the OpenAIV2 project

5852395

.Net OpenAI V2 - Internal Utilities - Phase 03 (#6970)

c8d9ade

- Updating policies using OpenAI SDK approach (GenericPolicy) impl. - Updated Unit Tests - Moved policy impl to openai Utilities.

Merge from main

c54127a

Add connector unit tests: Qdrant, Redis

43d7ecb

.Net: OpenAI V2 - Migrated FileService - Phase 05 (#7076)

caed23a

### Motivation and Context - Added FileService OpenAI V2 Implementation - Updated Extensions and UT accordingly - Updated ClientCore to be used with FileService and allow non-required `modelId`.

.Net: Clean-up (#7107)

5eefea7

### Motivation and Context A few cosmetic improvements were identified while migrating {Azure}OpenAI services to Azure.AI.OpenAI SDK v2. This PR implements those improvements.

github-actions bot changed the title ~~.Net: OpenAI V2 & Assistants V2 - GA~~ Python: .Net: OpenAI V2 & Assistants V2 - GA Sep 3, 2024

dmytrostruk removed the python Pull requests for the Python Semantic Kernel label Sep 3, 2024

dmytrostruk changed the title ~~Python: .Net: OpenAI V2 & Assistants V2 - GA~~ .Net: OpenAI V2 & Assistants V2 - GA Sep 3, 2024

markwallace-microsoft approved these changes Sep 3, 2024

View reviewed changes

markwallace-microsoft enabled auto-merge September 3, 2024 22:34

markwallace-microsoft added this pull request to the merge queue Sep 3, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 3, 2024

markwallace-microsoft added this pull request to the merge queue Sep 3, 2024

markwallace-microsoft removed this pull request from the merge queue due to a manual request Sep 3, 2024

.Net: [Feature Branch] Updated integration tests (#8500)

02d2208

markwallace-microsoft added the python Pull requests for the Python Semantic Kernel label Sep 3, 2024

github-actions bot changed the title ~~.Net: OpenAI V2 & Assistants V2 - GA~~ Python: .Net: OpenAI V2 & Assistants V2 - GA Sep 3, 2024

markwallace-microsoft approved these changes Sep 3, 2024

View reviewed changes

dmytrostruk enabled auto-merge September 3, 2024 23:27

dmytrostruk added this pull request to the merge queue Sep 3, 2024

markwallace-microsoft changed the title ~~Python: .Net: OpenAI V2 & Assistants V2 - GA~~ .Net: OpenAI V2 & Assistants V2 - GA Sep 3, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 3, 2024

Added service id to OpenAI text-to-image config

9c2ee04

dmytrostruk enabled auto-merge September 4, 2024 00:00

markwallace-microsoft approved these changes Sep 4, 2024

View reviewed changes

dmytrostruk added this pull request to the merge queue Sep 4, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 4, 2024

markwallace-microsoft added this pull request to the merge queue Sep 4, 2024

Merged via the queue into main with commit cc63d56 Sep 4, 2024
18 checks passed

markwallace-microsoft deleted the feature-connectors-openai branch September 4, 2024 00:45

crickman mentioned this pull request Sep 4, 2024

.Net: Bug Agents Assistants V2: Update usage of CancellationToken once Open AI SDK version is updated #7431

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

.Net: OpenAI V2 & Assistants V2 - GA #7151

.Net: OpenAI V2 & Assistants V2 - GA #7151

Uh oh!

crickman commented Jul 8, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arynaq commented Sep 4, 2024

Uh oh!

Uh oh!

.Net: OpenAI V2 & Assistants V2 - GA #7151

.Net: OpenAI V2 & Assistants V2 - GA #7151

Uh oh!

Conversation

crickman commented Jul 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arynaq commented Sep 4, 2024

Uh oh!

Uh oh!

crickman commented Jul 8, 2024 •

edited

Loading