Skip to content

Commit 9f1311a

Browse files
author
Preetam Joshi
committed
Updated postman collection. Added support for a user_query parameter in the input data dictionary.
1 parent 7feabec commit 9f1311a

File tree

3 files changed

+16
-4
lines changed

3 files changed

+16
-4
lines changed
Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,10 @@
11
{
22
"info": {
3-
"_postman_id": "81794dd9-392a-4646-9748-0b1ade43afbd",
3+
"_postman_id": "51d15140-83f0-44e6-99e4-cc31484a512e",
44
"name": "[Beta] Aimon APIs",
55
"description": "## Overview\n\nThis is a beta version of **Aimon Rely.** It includes our proprietary hallucination detector. This is an beta-release, so please treat it as such. Check with us (send a note to [info@aimon.ai](https://mailto:info@aimon.ai)) before using this API in a production setting. There are limited uptime guarantees at the moment. Please report any issues to the Aimon team (at [info@aimon.ai](https://mailto:info@aimon.ai)).\n\n> Use the APIs with caution - do not send sensitive or protected data to this API. \n \n\n## Features\n\n#### Hallucination detection\n\nGiven a context and the generated text, this API is able to detect 2 different types of model hallucinations: intrinsic and extrinsic.\n\n- The \"is_hallucinated\" field indicates whether the \"generated_text\" (passed in the input) is hallucinated.\n- A top level passage level \"score\" indicates if the entire set of sentences contain any hallucinations. The score is a probabilty measure of how hallucinated the text is compared to the context. A score >= 0.5 can be classified as a hallucination.\n- We also provide sentence level scores to help with explanability.\n \n\n#### Completeness detection\n\nGiven a context, generated text and optionally a reference text, this API is able to detect if the generated text completely answered the user's question. The context should include the context documents along with the user query as passed in to the LLM.\n\nThe output contains a \"score\" that is between 0.0 and 1.0 which indicates the degree of completeness. If the generated answer is not at all relevant to the user query, a score between 0.0 to 0.2 is possible. If the generated answer is relevant but misses some information, a score between 0.2 and 0.7 is possible. If the generated answer is relevant and fully captures all of the information, a score between 0.7 and 1.0 is possible.\n\nThe API also includes a \"reasoning\" field that is a text based explanation of the score. It also does a best effort method of pointing out the points that were missed from the expected answer.\n\n#### Conciseness detection\n\nGiven a context, generated text and optionally a reference text, this API is able to detect if the generated text was concise or verbose in terms of addressing the user query. The context should include the context documents along with the user query as passed in to the LLM.\n\nThe output contains a \"score\" that is between 0.0 and 1.0 which indicates the degree of conciseness. If the generated answer is very verbose and contains a lot of un-necessary information that is not relevant to the user query, a score between 0.0 to 0.2 is possible. If the generated answer is mostly relevant to the user query but has some amount of text that is not necessary for the user query a score between 0.2 and 0.7 is possible. If the generated answer is very concise and properly addresses all important points for the user query, a score between 0.7 and 1.0 is possible.\n\nThe API also includes a \"reasoning\" field that is a text based explanation of the score. It also does a best effort method of pointing out the un-necessary information that was included in the output.\n\n## **Limitations**\n\n- Input payloads with context sizes greater than 32,000 tokens will not work at the moment.\n- Maximum batch size is 25 items at the moment.",
66
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json",
7-
"_exporter_id": "30634528",
8-
"_collection_link": "https://aimon-trailblazers.postman.co/workspace/0c99cd4f-6ba5-41e9-9cbf-4f942a218086/collection/30634662-81794dd9-392a-4646-9748-0b1ade43afbd?action=share&source=collection_link&creator=30634528"
7+
"_exporter_id": "30634528"
98
},
109
"item": [
1110
{

src/aimon_rely_client/simple_client.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,7 @@ def detect(self, data_to_send: List[Dict[str, Any]]):
119119
"""
120120
Sends an HTTP POST request to the Aimon Rely Hallucination Detection API
121121
:param data_to_send: An array of dict objects where each dict contains a "context", a "generated_text" and
122-
optionally a "config" object
122+
optionally a "user_query" and "config" object
123123
:return: A JSON object containing the following fields (if applicable):
124124
"hallucination": Indicates whether the response consisted of intrinsic or extrinsic hallucinations.
125125
"is_hallucinated": top level string indicating if hallucinated or not,

src/aimon_rely_client/simple_client_test.py

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,19 @@ def test_valid_data_valid_response(self):
2222
assert len(response['hallucination']["sentences"]) == 1
2323
assert "This is the context" in response['hallucination']["sentences"][0]["text"]
2424

25+
def test_valid_data_valid_response_user_query(self):
26+
config = Config({'hallucination': 'default'})
27+
client = SimpleAimonRelyClient(api_key=API_KEY, config=config)
28+
data_to_send = [{"context": "This is the context", "user_query": "This is the user query", "generated_text": "This is the context"}]
29+
response = client.detect(data_to_send)[0]
30+
assert "hallucination" in response
31+
assert "is_hallucinated" in response['hallucination']
32+
assert response['hallucination']["is_hallucinated"] == "False"
33+
assert "score" in response['hallucination']
34+
assert "sentences" in response['hallucination']
35+
assert len(response['hallucination']["sentences"]) == 1
36+
assert "This is the context" in response['hallucination']["sentences"][0]["text"]
37+
2538
def test_valid_batch_data_valid_response(self):
2639
config = Config({'hallucination': 'default'})
2740
client = SimpleAimonRelyClient(api_key=API_KEY, config=config)

0 commit comments

Comments
 (0)