Updated postman collection. Added support for a user_query parameter in the input data dictionary.

Author: Preetam Joshi

postman_collections/aimon_hallucination_detection_beta.postman_collection.march2024.json renamed to postman_collections/aimon_apis_beta.postman_collection.may_2024.json

@@ -1,11 +1,10 @@
 {
 	"info": {
-		"_postman_id": "81794dd9-392a-4646-9748-0b1ade43afbd",
+		"_postman_id": "51d15140-83f0-44e6-99e4-cc31484a512e",
 		"name": "[Beta] Aimon APIs",
 		"description": "## Overview\n\nThis is a beta version of **Aimon Rely.** It includes our proprietary hallucination detector. This is an beta-release, so please treat it as such. Check with us (send a note to [info@aimon.ai](https://mailto:info@aimon.ai)) before using this API in a production setting. There are limited uptime guarantees at the moment. Please report any issues to the Aimon team (at [info@aimon.ai](https://mailto:info@aimon.ai)).\n\n> Use the APIs with caution - do not send sensitive or protected data to this API. \n  \n\n## Features\n\n#### Hallucination detection\n\nGiven a context and the generated text, this API is able to detect 2 different types of model hallucinations: intrinsic and extrinsic.\n\n- The \"is_hallucinated\" field indicates whether the \"generated_text\" (passed in the input) is hallucinated.\n- A top level passage level \"score\" indicates if the entire set of sentences contain any hallucinations. The score is a probabilty measure of how hallucinated the text is compared to the context. A score >= 0.5 can be classified as a hallucination.\n- We also provide sentence level scores to help with explanability.\n    \n\n#### Completeness detection\n\nGiven a context, generated text and optionally a reference text, this API is able to detect if the generated text completely answered the user's question. The context should include the context documents along with the user query as passed in to the LLM.\n\nThe output contains a \"score\" that is between 0.0 and 1.0 which indicates the degree of completeness. If the generated answer is not at all relevant to the user query, a score between 0.0 to 0.2 is possible. If the generated answer is relevant but misses some information, a score between 0.2 and 0.7 is possible. If the generated answer is relevant and fully captures all of the information, a score between 0.7 and 1.0 is possible.\n\nThe API also includes a \"reasoning\" field that is a text based explanation of the score. It also does a best effort method of pointing out the points that were missed from the expected answer.\n\n#### Conciseness detection\n\nGiven a context, generated text and optionally a reference text, this API is able to detect if the generated text was concise or verbose in terms of addressing the user query. The context should include the context documents along with the user query as passed in to the LLM.\n\nThe output contains a \"score\" that is between 0.0 and 1.0 which indicates the degree of conciseness. If the generated answer is very verbose and contains a lot of un-necessary information that is not relevant to the user query, a score between 0.0 to 0.2 is possible. If the generated answer is mostly relevant to the user query but has some amount of text that is not necessary for the user query a score between 0.2 and 0.7 is possible. If the generated answer is very concise and properly addresses all important points for the user query, a score between 0.7 and 1.0 is possible.\n\nThe API also includes a \"reasoning\" field that is a text based explanation of the score. It also does a best effort method of pointing out the un-necessary information that was included in the output.\n\n## **Limitations**\n\n- Input payloads with context sizes greater than 32,000 tokens will not work at the moment.\n- Maximum batch size is 25 items at the moment.",
 		"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json",
-		"_exporter_id": "30634528",
-		"_collection_link": "https://aimon-trailblazers.postman.co/workspace/0c99cd4f-6ba5-41e9-9cbf-4f942a218086/collection/30634662-81794dd9-392a-4646-9748-0b1ade43afbd?action=share&source=collection_link&creator=30634528"
+		"_exporter_id": "30634528"
 	},
 	"item": [
 		{

src/aimon_rely_client/simple_client.py

@@ -119,7 +119,7 @@ def detect(self, data_to_send: List[Dict[str, Any]]):
         """
         Sends an HTTP POST request to the Aimon Rely Hallucination Detection API
         :param data_to_send: An array of dict objects where each dict contains a "context", a "generated_text" and
-                             optionally a "config" object
+                             optionally a "user_query" and "config" object
         :return: A JSON object containing the following fields (if applicable):
                 "hallucination": Indicates whether the response consisted of intrinsic or extrinsic hallucinations.
                     "is_hallucinated": top level string indicating if hallucinated or not,

src/aimon_rely_client/simple_client_test.py

@@ -22,6 +22,19 @@ def test_valid_data_valid_response(self):
         assert len(response['hallucination']["sentences"]) == 1
         assert "This is the context" in response['hallucination']["sentences"][0]["text"]
 
+    def test_valid_data_valid_response_user_query(self):
+        config = Config({'hallucination': 'default'})
+        client = SimpleAimonRelyClient(api_key=API_KEY, config=config)
+        data_to_send = [{"context": "This is the context", "user_query": "This is the user query", "generated_text": "This is the context"}]
+        response = client.detect(data_to_send)[0]
+        assert "hallucination" in response
+        assert "is_hallucinated" in response['hallucination']
+        assert response['hallucination']["is_hallucinated"] == "False"
+        assert "score" in response['hallucination']
+        assert "sentences" in response['hallucination']
+        assert len(response['hallucination']["sentences"]) == 1
+        assert "This is the context" in response['hallucination']["sentences"][0]["text"]
+
     def test_valid_batch_data_valid_response(self):
         config = Config({'hallucination': 'default'})
         client = SimpleAimonRelyClient(api_key=API_KEY, config=config)