|
| 1 | +# Azure Online Experimentation client library for Python |
| 2 | + |
| 3 | +This package contains Azure Online Experimentation client library for interacting with `Microsoft.OnlineExperimentation/workspaces` resources. |
| 4 | + |
| 5 | +## Getting started |
| 6 | + |
| 7 | +### Install the package |
| 8 | + |
| 9 | +```bash |
| 10 | +python -m pip install azure-onlineexperimentation |
| 11 | +``` |
| 12 | + |
| 13 | +#### Prequisites |
| 14 | + |
| 15 | +- Python 3.9 or later is required to use this package. |
| 16 | +- You need an [Azure subscription][azure_sub] to use this package. |
| 17 | +- An [Azure Online Experimentation workspace][azure_exp_workspace] resource in the Azure subscription. |
| 18 | + |
| 19 | +### Create and authenticate the client |
| 20 | + |
| 21 | +The Azure Online Experimentation client library initialization requires two parameters: |
| 22 | + |
| 23 | +- The `endpoint` property value from the [`Microsoft.OnlineExperimentation/workspaces`][azure_exp_workspace] resource. |
| 24 | +- A credential from `azure.identity`, the simplest approach is to use [DefaultAzureCredential][default_azure_credential] and `az login` to authenticate. See [Azure Identity client library for Python][azure_identity_credentials] for more details. |
| 25 | + |
| 26 | +To construct a synchronous client: |
| 27 | + |
| 28 | +<!-- SNIPPET:sample_initialize_client.initialize_client --> |
| 29 | + |
| 30 | +```python |
| 31 | +import os |
| 32 | +from azure.identity import DefaultAzureCredential |
| 33 | +from azure.onlineexperimentation import OnlineExperimentationClient |
| 34 | + |
| 35 | +# Create a client with your Online Experimentation workspace endpoint and credentials |
| 36 | +endpoint = os.environ["AZURE_ONLINEEXPERIMENTATION_ENDPOINT"] |
| 37 | +client = OnlineExperimentationClient(endpoint, DefaultAzureCredential()) |
| 38 | +print(f"Client initialized with endpoint: {endpoint}") |
| 39 | +``` |
| 40 | + |
| 41 | +<!-- END SNIPPET --> |
| 42 | + |
| 43 | +To construct an asynchronous client, instead import `OnlineExperimentationClient` from `azure.onlineexperimentation.aio` and `DefaultAzureCredential` from `azure.identity.aio` namespaces: |
| 44 | + |
| 45 | +<!-- SNIPPET:sample_initialize_async_client.initialize_async_client --> |
| 46 | + |
| 47 | +```python |
| 48 | +import os |
| 49 | +from azure.identity.aio import DefaultAzureCredential |
| 50 | +from azure.onlineexperimentation.aio import OnlineExperimentationClient |
| 51 | + |
| 52 | +# Create a client with your Online Experimentation workspace endpoint and credentials |
| 53 | +endpoint = os.environ["AZURE_ONLINEEXPERIMENTATION_ENDPOINT"] |
| 54 | +client = OnlineExperimentationClient(endpoint, DefaultAzureCredential()) |
| 55 | +print(f"Client initialized with endpoint: {endpoint}") |
| 56 | +``` |
| 57 | + |
| 58 | +<!-- END SNIPPET --> |
| 59 | + |
| 60 | +## Key concepts |
| 61 | + |
| 62 | +### Online Experimentation Workspace |
| 63 | + |
| 64 | +[`Microsoft.OnlineExperimentation/workspaces`][az_exp_workspace] Azure resources work in conjunction with [Azure App Configuration][app_config] and [Azure Monitor][azure_monitor]. The Online Experimentation workspace handles management of metrics definitions and their continuous computation to monitor and evaluate experiment results. |
| 65 | + |
| 66 | +### Experiment Metrics |
| 67 | + |
| 68 | +Metrics are used to measure the impact of your online experiments. See the [samples][azure_exp_samples] for how to create and manage various types of experiment metrics. |
| 69 | + |
| 70 | +## Troubleshooting |
| 71 | + |
| 72 | +Errors can occur during initial requests and will provide information about how to resolve the error. |
| 73 | + |
| 74 | +## Examples |
| 75 | + |
| 76 | +This examples goes theough the experiment metric management lifecycle, to run the example: |
| 77 | + |
| 78 | +- Set `AZURE_ONLINEEXPERIMENTATION_ENDPOINT` environment variable to the `endpoint` property value (URL) from a [`Microsoft.OnlineExperimentation/workspaces`][az_exp_workspace] resource. |
| 79 | +- Enable `DefaultAzureCredential` by running `az login` or `Connect-AzAccount`, see [documentation][default_azure_credential] for details and troubleshooting. |
| 80 | + |
| 81 | +<!-- SNIPPET:sample_experiment_metrics_management.experiment_metrics_management --> |
| 82 | + |
| 83 | +```python |
| 84 | +import os |
| 85 | +import random |
| 86 | +import json |
| 87 | +from azure.identity import DefaultAzureCredential |
| 88 | +from azure.onlineexperimentation import OnlineExperimentationClient |
| 89 | +from azure.onlineexperimentation.models import ( |
| 90 | + ExperimentMetric, |
| 91 | + LifecycleStage, |
| 92 | + DesiredDirection, |
| 93 | + UserRateMetricDefinition, |
| 94 | + ObservedEvent, |
| 95 | +) |
| 96 | +from azure.core.exceptions import HttpResponseError |
| 97 | + |
| 98 | +# [Step 1] Initialize the SDK client |
| 99 | +# The endpoint URL from the Microsoft.OnlineExperimentation/workspaces resource |
| 100 | +endpoint = os.environ.get("AZURE_ONLINEEXPERIMENTATION_ENDPOINT", "<endpoint-not-set>") |
| 101 | +credential = DefaultAzureCredential() |
| 102 | + |
| 103 | +print(f"AZURE_ONLINEEXPERIMENTATION_ENDPOINT is {endpoint}") |
| 104 | + |
| 105 | +client = OnlineExperimentationClient(endpoint=endpoint, credential=credential) |
| 106 | + |
| 107 | +# [Step 2] Define the experiment metric |
| 108 | +example_metric = ExperimentMetric( |
| 109 | + lifecycle=LifecycleStage.ACTIVE, |
| 110 | + display_name="% users with LLM interaction who made a high-value purchase", |
| 111 | + description="Percentage of users who received a response from the LLM and then made a purchase of $100 or more", |
| 112 | + categories=["Business"], |
| 113 | + desired_direction=DesiredDirection.INCREASE, |
| 114 | + definition=UserRateMetricDefinition( |
| 115 | + start_event=ObservedEvent(event_name="ResponseReceived"), |
| 116 | + end_event=ObservedEvent(event_name="Purchase", filter="Revenue > 100"), |
| 117 | + ) |
| 118 | +) |
| 119 | + |
| 120 | +# [Optional][Step 2a] Validate the metric - checks for input errors without persisting anything |
| 121 | +print("Checking if the experiment metric definition is valid...") |
| 122 | +print(json.dumps(example_metric.as_dict(), indent=2)) |
| 123 | + |
| 124 | +try: |
| 125 | + validation_result = client.validate_metric(example_metric) |
| 126 | + |
| 127 | + print(f"Experiment metric definition valid: {validation_result.is_valid}.") |
| 128 | + for detail in validation_result.diagnostics or []: |
| 129 | + # Inspect details of why the metric definition was rejected as Invalid |
| 130 | + print(f"- {detail.code}: {detail.message}") |
| 131 | + |
| 132 | + # [Step 3] Create the experiment metric |
| 133 | + example_metric_id = f"sample_metric_id_{random.randint(10000, 20000)}" |
| 134 | + |
| 135 | + print(f"Creating the experiment metric {example_metric_id}...") |
| 136 | + # Using upsert to create the metric with If-None-Match header |
| 137 | + create_response = client.create_or_update_metric( |
| 138 | + experiment_metric_id=example_metric_id, |
| 139 | + resource=example_metric, |
| 140 | + match_condition=None, # This ensures If-None-Match: * header is sent |
| 141 | + etag=None |
| 142 | + ) |
| 143 | + |
| 144 | + print(f"Experiment metric {create_response.id} created, etag: {create_response.e_tag}.") |
| 145 | + |
| 146 | + # [Step 4] Deactivate the experiment metric and update the description |
| 147 | + updated_metric = { |
| 148 | + "lifecycle": LifecycleStage.INACTIVE, # pauses computation of this metric |
| 149 | + "description": "No longer need to compute this." |
| 150 | + } |
| 151 | + |
| 152 | + update_response = client.create_or_update_metric( |
| 153 | + experiment_metric_id=example_metric_id, |
| 154 | + resource=updated_metric, |
| 155 | + etag=create_response.e_tag, # Ensures If-Match header is sent |
| 156 | + match_condition=None # Not specifying match_condition as we're using etag |
| 157 | + ) |
| 158 | + |
| 159 | + print(f"Updated metric: {update_response.id}, etag: {update_response.e_tag}.") |
| 160 | + |
| 161 | + # [Step 5] Delete the experiment metric |
| 162 | + client.delete_metric( |
| 163 | + experiment_metric_id=example_metric_id, |
| 164 | + etag=update_response.e_tag # Ensures If-Match header is sent |
| 165 | + ) |
| 166 | + |
| 167 | + print(f"Deleted metric: {example_metric_id}.") |
| 168 | + |
| 169 | +except HttpResponseError as error: |
| 170 | + print(f"The operation failed with error: {error}") |
| 171 | +``` |
| 172 | + |
| 173 | +<!-- END SNIPPET --> |
| 174 | + |
| 175 | +## Next steps |
| 176 | + |
| 177 | +Have a look at the [samples][azure_exp_samples] folder, containing fully runnable Python code for synchronous and asynchronous clients. |
| 178 | + |
| 179 | +## Contributing |
| 180 | + |
| 181 | +This project welcomes contributions and suggestions. Most contributions require |
| 182 | +you to agree to a Contributor License Agreement (CLA) declaring that you have |
| 183 | +the right to, and actually do, grant us the rights to use your contribution. |
| 184 | +For details, visit <https://cla.microsoft.co>m. |
| 185 | + |
| 186 | +When you submit a pull request, a CLA-bot will automatically determine whether |
| 187 | +you need to provide a CLA and decorate the PR appropriately (e.g., label, |
| 188 | +comment). Simply follow the instructions provided by the bot. You will only |
| 189 | +need to do this once across all repos using our CLA. |
| 190 | + |
| 191 | +This project has adopted the |
| 192 | +[Microsoft Open Source Code of Conduct][code_of_conduct]. For more information, |
| 193 | +see the Code of Conduct FAQ or contact <opencode@microsoft.com> with any |
| 194 | +additional questions or comments. |
| 195 | + |
| 196 | +<!-- LINKS --> |
| 197 | +[code_of_conduct]: https://opensource.microsoft.com/codeofconduct/ |
| 198 | +[azure_identity_credentials]: https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/identity/azure-identity#credentials |
| 199 | + |
| 200 | +[default_azure_credential]: https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/identity/azure-identity#defaultazurecredential |
| 201 | +[azure_sub]: https://azure.microsoft.com/free/ |
| 202 | +[azure_exp_workspace]: https://learn.microsoft.com/azure/templates/microsoft.onlineexperimentation/workspaces |
| 203 | +[app_config]: https://learn.microsoft.com/azure/azure-app-configuration/overview |
| 204 | +[azure_monitor]: https://learn.microsoft.com/azure/azure-monitor/overview |
| 205 | +[azure_exp_samples]: https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/onlineexperimentation/azure-onlineexperimentation/samples/ |
0 commit comments