Externalize the LLM Prompts.

[DanHUMassMed]

I would like to suggest a new feature. Externalize the LLM Prompts.

A number of User Stories I see positively impacted are:

• As a user, I want to tweak certain prompts for specific LLMs. This will improve the research outcomes and avoid code exceptions for different models/providers without requiring code changes.
• As a user, I want to customize the “choose agent” prompt for a specific set of research questions/Use Cases, which would enable more focused and accurate responses from the LLM for the specific need.
• As a user, I want to modify the responses to certain report prompts. This will create a more customized report that caters to my Use Case.
• As a user, I want to do A-B testing on certain prompts, enabling human feedback to inform/enhance prompts.

Some Ideation on Implementation details.

• A default set of prompts is maintained as a resource (prompts.json), and no config variable is needed.
• If a config variable is defined, it is a path to a JSON file that will override the defaults.
• The JSON maps to a parameterized string.
• The JSON implementation is maintained as an array of strings (To enhance readability and maintenance).

{
  "search_queries_prompt":[
    "Write {max_iterations} Google search queries to search online that form an objective opinion from the following task: {task}\n",
    "Use the current date if needed: {datetime_now}.\n",
    "Also include in the queries specified task details such as locations, names, etc.\n",
    "You MUST respond with a list of strings in the following format: [\"query 1\", \"query 2\", \"query 3\"].",
    "The response should contain ONLY the list."
    ],
  "research_report_prompt":[
    "Information: '''{context}'''\n\n",
    "Using the above information, answer the following query or task: '{question}' in a detailed report -- ",
    "The report should focus on the answer to the query, should be well structured, informative, ",
    "in-depth and comprehensive, with facts and numbers if available and a minimum of {total_words} words.\n",
    "You should strive to write the report as long as you can using all relevant and necessary information provided.\n",
    "You must write the report with markdown syntax.\n",
    "Use an unbiased and journalistic tone.\n",
    . . .
}

Discussion/Ideas

• Would like to here comments from folks on if the idea is solid and fits the future direction.
• Does this align with any other current or planned work and thus should be coordinated?
• What are some of the potential pitfalls of this approach?
• What are some other potentially better implementation ideas?

Thanks,
Dan