ref(feedback): better prompt for AI summaries, reduced LLM temperature (#93997)

Author: vishnupsatish

src/sentry/feedback/usecases/feedback_summaries.py

@@ -13,9 +13,11 @@ def make_input_prompt(
     feedbacks_string = "\n------\n".join(feedbacks)
     return f"""Instructions:
 
-You are an assistant that summarizes customer feedback. Given a list of customer feedback entries, generate a concise summary of 1-2 sentences that reflects the key themes. Begin the summary with "Users...", for example, "Users say...". Don't make overly generic statements like "Users report a variety of issues."
+You are an assistant that summarizes customer feedback. Given a list of customer feedback entries, generate a concise summary of 1-2 sentences where you summarize the broader themes that represent the list of feedbacks that you are given.
 
-Balance specificity and generalization based on the size of the input and based only on the themes and topics present in the list of customer feedback entries. Your goal is to focus on identifying and summarizing broader themes that are mentioned more frequently across different feedback entries. For example, if there are many feedback entries, it makes more sense to prioritize mentioning broader themes that apply to many feedbacks, versus mentioning one or two specific isolated concerns and leaving out others that are just as prevalent.
+Your goal is to help someone quickly understand the main patterns or themes that represent the feedback entries. Don't mention ANY specific examples of themes. The summary should remain broad.
+
+Begin the summary with "Users...", for example, "Users say...". Don't make overly generic statements like "Users report a variety of issues."
 
 The summary must be AT MOST 55 words, that is an absolute upper limit, and you must write AT MOST two sentences. You can leave certain things out, and when deciding what topics/themes to mention, make sure it is proportional to the number of times they appear in different customer feedback entries.
 
@@ -36,7 +38,7 @@ def generate_summary(
     response = complete_prompt(  # This can throw
         usecase=LLMUseCase.FEEDBACK_SUMMARIES,
         message=make_input_prompt(feedbacks),
-        temperature=0.3,
+        temperature=0.1,
         max_output_tokens=150,
     )