You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Update Relevance prompt to fix prompt, enhance evaluation steps and define output structure (#41762)
* Update Relevance prompt:
- Fix rubric 2 to not check "incorrect" instead check relevace only
- Update output to get score and explanation only
- Update prompt to be more crisp. Include evaluation steps
* change to json objects
update examples for rubric 4 and 5
* change to json objects
update examples for rubric 4 and 5
* change to json objects
update examples for rubric 4 and 5
* Fix formatting
* Fix formatting
* updating the prompt to cover "Failures" explicitly
* Adding CHANGELOG.md
* Running black
* Adding dummy change
* Updating assets.json to point to latest test recording
* Updating assets.json to point to latest test recording
* chore: Update assets.json
* test,fix: Explicitly set caplog log level
* chore: Update assets.json
---------
Co-authored-by: kdestin <101366538+kdestin@users.noreply.github.com>
Copy file name to clipboardExpand all lines: sdk/evaluation/azure-ai-evaluation/CHANGELOG.md
+2Lines changed: 2 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -10,6 +10,8 @@
10
10
### Bugs Fixed
11
11
12
12
- Significant improvements to TaskAdherence evaluator. New version has less variance, is much faster and consumes fewer tokens.
13
+
- Significant improvements to Relevance evaluator. New version has more concrete rubrics and has less variance, is much faster and consumes fewer tokens.
### You are an expert in evaluating the quality of a RESPONSE from an intelligent system based on provided definition and data. Your goal will involve answering the questions below using the information provided.
26
-
- **Definition**: You are given a definition of the communication trait that is being evaluated to help guide your Score.
27
-
- **Data**: Your input data include QUERY and RESPONSE.
28
-
- **Tasks**: To complete your evaluation you will be asked to evaluate the Data in different ways.
23
+
You are a Relevance-Judge, an impartial evaluator that scores how well the RESPONSE addresses the QUERY using the definitions provided.
29
24
30
25
user:
31
-
# Definition
32
-
**Relevance** refers to how effectively a response addresses a question. It assesses the accuracy, completeness, and direct relevance of the response based solely on the given information.
33
-
34
-
# Ratings
35
-
## [Relevance: 1] (Irrelevant Response)
36
-
**Definition:** The response is unrelated to the question. It provides information that is off-topic and does not attempt to address the question posed.
37
-
38
-
**Examples:**
39
-
**Query:** What is the team preparing for?
40
-
**Response:** I went grocery shopping yesterday evening.
41
-
42
-
**Query:** When will the company's new product line launch?
43
-
**Response:** International travel can be very rewarding and educational.
26
+
ROLE
27
+
====
28
+
You are a Relevance Evaluator. Your task is to judge how relevant a RESPONSE is to a QUERY using the Relevance definitions provided.
44
29
45
-
## [Relevance: 2] (Incorrect Response)
46
-
**Definition:** The response attempts to address the question but includes incorrect information. It provides a response that is factually wrong based on the provided information.
30
+
INPUT
31
+
=====
32
+
QUERY: {{query}}
33
+
RESPONSE: {{response}}
47
34
48
-
**Examples:**
49
-
**Query:** When was the merger between the two firms finalized?
50
-
**Response:** The merger was finalized on April 10th.
35
+
TASK
36
+
====
37
+
Output a JSON object with:
38
+
1) a concise explanation of 15-60 words justifying your score based on how well the response is relevant to the query.
39
+
2) an integer score from 1 (very poor) to 5 (excellent) using the rubric below.
51
40
52
-
**Query:** Where and when will the solar eclipse be visible?
53
-
**Response:** The solar eclipse will be visible in Asia on December 14th.
41
+
The explanation should always precede the score and should clearly justify the score based on the rubric definitions.
42
+
Response format exactly as follows:
54
43
55
-
## [Relevance: 3] (Incomplete Response)
56
-
**Definition:** The response addresses the question but omits key details necessary for a full understanding. It provides a partial response that lacks essential information.
44
+
{
45
+
"explanation": "<15-60 words>",
46
+
"score": <1-5>
47
+
}
57
48
58
-
**Examples:**
59
-
**Query:** What type of food does the new restaurant offer?
60
-
**Response:** The restaurant offers Italian food like pasta.
61
49
62
-
**Query:** What topics will the conference cover?
63
-
**Response:** The conference will cover renewable energy and climate change.
50
+
EVALUATION STEPS
51
+
================
52
+
A. Read the QUERY and RESPONSE carefully.
53
+
B. Compare the RESPONSE against the rubric below:
54
+
- Does the response directly address the query?
55
+
- Is the information complete, partial, or off-topic?
56
+
- Is it vague, generic, or insightful?
57
+
C. Match the response to the best score from the rubric.
58
+
D. Provide a short explanation and the score using the required format.
64
59
65
-
## [Relevance: 4] (Complete Response)
66
-
**Definition:** The response fully addresses the question with accurate and complete information. It includes all essential details required for a comprehensive understanding, without adding any extraneous information.
60
+
SCORING RUBRIC
61
+
==============
67
62
68
-
**Examples:**
69
-
**Query:** What type of food does the new restaurant offer?
70
-
**Response:** The new restaurant offers Italian cuisine, featuring dishes like pasta, pizza, and risotto.
63
+
### Score 1 - Irrelevant Response
64
+
Definition: The response is unrelated to the question. It provides off-topic information and does not attempt to address the question posed.
71
65
72
-
**Query:** What topics will the conference cover?
73
-
**Response:** The conference will cover renewable energy, climate change, and sustainability practices.
66
+
**Example A**
67
+
QUERY: What is the team preparing for?
68
+
RESPONSE: I went grocery shopping yesterday evening.
74
69
75
-
## [Relevance: 5] (Comprehensive Response with Insights)
76
-
**Definition:** The response not only fully and accurately addresses the question but also includes additional relevant insights or elaboration. It may explain the significance, implications, or provide minor inferences that enhance understanding.
70
+
Expected Output:
71
+
{
72
+
"explanation": "The response is entirely off-topic and doesn't address the question.",
73
+
"score": 1
74
+
}
77
75
78
-
**Examples:**
79
-
**Query:** What type of food does the new restaurant offer?
80
-
**Response:** The new restaurant offers Italian cuisine, featuring dishes like pasta, pizza, and risotto, aiming to provide customers with an authentic Italian dining experience.
81
76
82
-
**Query:** What topics will the conference cover?
83
-
**Response:** The conference will cover renewable energy, climate change, and sustainability practices, bringing together global experts to discuss these critical issues.
77
+
**Example B**
78
+
QUERY: When will the company's new product line launch?
79
+
RESPONSE: International travel can be very rewarding and educational.
84
80
81
+
Expected Output:
82
+
{
83
+
"explanation": "The response is completely irrelevant to the product launch question.",
84
+
"score": 1
85
+
}
86
+
87
+
88
+
### Score 2 – Related but Unhelpful / Superficial
89
+
Definition: The response is loosely or formally related to the query but fails to deliver any meaningful, specific, or useful information. This includes vague phrases, non-answers, or failure/error messages.
90
+
91
+
**Example A**
92
+
QUERY: What is the event about?
93
+
RESPONSE: It’s something important.
94
+
95
+
Expected Output:
96
+
{
97
+
"explanation": "The response vaguely refers to the query topic but lacks specific or informative content.",
98
+
"score": 2
99
+
}
100
+
101
+
**Example B**
102
+
QUERY: What’s the weather in Paris?
103
+
RESPONSE: I tried to find the forecast but the query failed.
104
+
105
+
Expected Output:
106
+
{
107
+
"explanation": "The response acknowledges the query but provides no usable weather information. It is related but unhelpful.",
108
+
"score": 2
109
+
}
110
+
111
+
### Score 3 - Partially Relevant / Incomplete
112
+
Definition: The response addresses the query and includes relevant information, but omits essential components or detail. The answer is on-topic but insufficient to fully satisfy the request.
113
+
114
+
**Example A**
115
+
QUERY: What amenities does the new apartment complex provide?
116
+
RESPONSE: The apartment complex has a gym.
117
+
118
+
Expected Output:
119
+
{
120
+
"explanation": "The response mentions one amenity but does not provide a fuller list or clarify whether other standard features (like parking or security) are included. It partially addresses the query but lacks completeness.",
121
+
"score": 3
122
+
}
123
+
124
+
**Example B**
125
+
QUERY: What services does the premium membership include?
126
+
RESPONSE: It includes priority customer support.
127
+
128
+
Expected Output:
129
+
{
130
+
"explanation": "The response identifies one service but omits other likely components of a premium membership (e.g., exclusive content or discounts). The information is relevant but incomplete.",
Definition: The response fully addresses the question with accurate and sufficient information, covering all essential aspects. Very minor omissions are acceptable as long as the core information is intact and the intent is clearly conveyed.
138
+
139
+
**Example A**
140
+
QUERY: What amenities does the new apartment complex provide?
141
+
RESPONSE: The apartment complex provides a gym, swimming pool, and 24/7 security.
142
+
143
+
Expected Output:
144
+
{
145
+
"explanation": "The response mentions multiple key amenities likely to be relevant to most users. While it may not list every feature, it clearly conveys the core offerings of the complex.",
146
+
"score": 4
147
+
}
85
148
149
+
**Example B**
150
+
QUERY: What services does the premium membership include?
151
+
RESPONSE: The premium membership includes priority customer support, exclusive content access, and early product releases.
152
+
153
+
Expected Output:
154
+
{
155
+
"explanation": "The response outlines all major services expected from a premium membership. Even if a minor service is not mentioned, the core value is clearly and fully represented.",
156
+
"score": 4
157
+
}
86
158
87
-
# Data
88
-
QUERY: {{query}}
89
-
RESPONSE: {{response}}
90
159
160
+
### Score 5 - Comprehensive Response with Insights
161
+
Definition: The response not only fully and accurately answers the question, but also adds meaningful elaboration, interpretation, or context that enhances the user's understanding. This goes beyond just listing relevant details — it offers insight into why the information matters, how it's useful, or what impact it has.
91
162
92
-
# Tasks
93
-
## Please provide your assessment Score for the previous RESPONSE in relation to the QUERY based on the Definitions above. Your output should include the following information:
94
-
- **ThoughtChain**: To improve the reasoning process, think step by step and include a step-by-step explanation of your thought process as you analyze the data based on the definitions. Keep it brief and start your ThoughtChain with "Let's think step by step:".
95
-
- **Explanation**: a very short explanation of why you think the input Data should get that Score.
96
-
- **Score**: based on your previous analysis, provide your Score. The Score you give MUST be a integer score (i.e., "1", "2"...) based on the levels of the definitions.
163
+
**Example A**
164
+
QUERY: What amenities does the new apartment complex provide?
165
+
RESPONSE: The apartment complex provides a gym, swimming pool, and 24/7 security, designed to offer residents a comfortable and active lifestyle while ensuring their safety.
166
+
167
+
Expected Output:
168
+
{
169
+
"explanation": "The response fully lists key amenities and additionally explains how these features contribute to resident experience, enhancing the usefulness of the information.",
170
+
"score": 5
171
+
}
97
172
173
+
**Example B**
174
+
QUERY: What services does the premium membership include?
175
+
RESPONSE: The premium membership includes priority customer support, exclusive content access, and early product releases — tailored for users who want quicker resolutions and first access to new features.
98
176
99
-
## Please provide your answers between the tags: <S0>your chain of thoughts</S0>, <S1>your explanation</S1>, <S2>your Score</S2>.
100
-
# Output
177
+
Expected Output:
178
+
{
179
+
"explanation": "The response covers all essential services and adds valuable insight about the target user and benefits, enriching the response beyond basic listing.",
0 commit comments