Azure AI Foundry
-
|
Hello, we're setting up evaluation in our project by following this tutorial: https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/agent-evaluate-sdk We see different behaviors if we target a hub-based project vs a Foundry project. When using a hub-based project, we get results in Azure AI Foundry in a nice table view that shows for each item the value of the computed metrics. When using a Foundry project, we need to have an Azure ML Workspace available with the same name as the Foundry project in order to be able to upload results. What's the recommended way to have results visible in Azure AI Foundry when using a Foundry project? |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments 6 replies
-
|
Hi @mzat-msft This is good feedback. The Foundry project capabilities are being constantly updated to reflect support that was previously in Hub. In this specific instance, let me check with the right team to get you a valid response on the status. In the meantime, these are the two resources I am sharing for clarity on Hub vs. Foundry.
|
Beta Was this translation helpful? Give feedback.
-
|
Hi @nitya, Do you have any updates? Another issue we encountered is that when integrating with Azure ML, the evaluation upload only succeeds if the storage account used by the ML workspace has Allow storage account key access enabled. This poses a problem for us, as our security policy requires that storage account key access be disabled. |
Beta Was this translation helpful? Give feedback.
-
|
@LauraDamianTNA thanks for the feedback! Let me check in with the team again - had made the request before the long weekend but not heard back yet. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @nitya, Regarding access to the storage account, I came across a preview feature that would help resolve the issue. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the nudge @LauraDamianTNA - the response below (@luigiw) is from the product team. If you can add more context to that reply, that would be good. And also thanks so much for doing the spadework on this and raising the issue. |
Beta Was this translation helpful? Give feedback.
-
|
Thx for the great feedback. The storage account gap is a known issue and will be bridged soon with managed storage support. It not clear to me what do you mean by below, you shouldn't need to create an Azure ML workspace to use a Foundry project, is this UI gap? " |
Beta Was this translation helpful? Give feedback.
-
|
This is the error stack trace that points to a requirement to have an Azure ML workspace. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the detailed traces and feedback @LauraDamianTNA @mzat-msft - I had a similar issue reported in a different thread so taking these as a single issue (related to dependency on Azure ML workspace) when using the Foundry project resource There is a post where I was tracking FDP related issues and sounds like we need more guidance for the Evaluations sample. |
Beta Was this translation helpful? Give feedback.
-
|
Thx for the sharing the stacktrace. I think what's happening here is some mis configuration when using the new AI Foundry projects. The error message was out of date, and the intention was not for you to create an AzureML workspace. Can you share the code how you configured to use the AI Foundry project? |
Beta Was this translation helpful? Give feedback.
-
|
Hello @luigiw Thank you for great pointer which helped us fixing the issue! I noticed we were configuring the This is what we had: evaluation_result = evaluate(
data=dataset_filename,
evaluation_name=f"eval-agent-{branch}-{commit[:7]}",
evaluators=evaluators,
evaluator_config={
"response_completeness": {
"column_mapping": {
"ground_truth": "${data.expected_information}",
"response": "${data.response}",
}
},
"information_retrieval": {
"column_mapping": {
"response": "${data.response}",
"relevant_documents": "${data.relevant_documents}",
}
},
},
azure_ai_project={
"subscription_id": env_cfg["azure_subscription_id"],
"project_name": env_cfg["azure_project_name"],
"resource_group_name": env_cfg["azure_resource_group"],
},
credential=AzureCliCredential(),
)This is how we fixed it: evaluation_result = evaluate(
data=dataset_filename,
evaluation_name=f"eval-agent-{branch}-{commit[:7]}",
evaluators=evaluators,
evaluator_config={
"response_completeness": {
"column_mapping": {
"ground_truth": "${data.expected_information}",
"response": "${data.response}",
}
},
"information_retrieval": {
"column_mapping": {
"response": "${data.response}",
"relevant_documents": "${data.relevant_documents}",
}
},
},
azure_ai_project=env_cfg["project_endpoint"],
credential=AzureCliCredential(),
) |
Beta Was this translation helpful? Give feedback.
Hello @luigiw
Thank you for great pointer which helped us fixing the issue!
I noticed we were configuring the
azure_ai_projectdifferently from the tutorial.I think we landed to that value after multiple experiment trying to make it work for the Hub and for the Foundry Project.
This is what we had: