Discussion about controlled LLM (guidance/lmql)

[caillef]

It seems that the idea right now is to make no difference between agents, from the top to the bottom, each agent is a simple "preprompt + instruction -> result" + code execution

After looking at indydevdan channel, I discovered guidance and lmql

I tested guidance to generate a first simple function and then add reflection/improvements.

import guidance

gpt4 = guidance.llms.OpenAI("gpt-4")

experts = guidance('''
{{#system~}}
You are a Python senior developer.
{{~/system}}

{{#user~}}
{{query}}
Write only the function using python. No comments.
{{~/user}}

{{#assistant~}}
{{gen 'function' temperature=0 max_tokens=500}}
{{~/assistant}}

{{#user~}}
Now reflect on your code and make it better (comments, variable names, loops)
{{~/user}}

{{#assistant~}}
{{gen 'final_function' temperature=0 max_tokens=500}}
{{~/assistant}}

''', llm=gpt4)

query = "write a function that list files in a specified directory and write the first line and the last line of the second file."
response = experts(query=query)

print(response["function"])
print(response["final_function"])

Why I think it is useful to use this kind of tool instead of classic agent completion

• Results are stored in variables, it can be in lists (no need to parse)
• We can easily call functions inside
• Costs seems lower than making many agents (need to test this), here it is just one agent that can use its own generation as variables.
• We can define deterministic output (here it will always select 9 or 10 or 11, nothing more or less, removing noise)

{{#select 'chapter'}}9{{or}}10{{or}}11{{/select}}

• We can easily switch to other models and we are not tied up to OpenAI (maybe not important right now but it would be cool if we can try local models without rewriting everything)

Now there is one good news and one bad news

• The bad news is that guidance has many issues right now and main is abandoned. But the team is working on a new branch, they are revamping everything from scratch.
• The good news is that we could make an agent to generate this kind of language. Here is what I have in mind:

Vision
We want to build an agent that makes perfect python function for our tools.

Current version:

• generate python function
• need to test it manually

Version using guidance:

• We have a first agent that generates "what are the steps to achieve this task the best way"
• The agent answer : "Generate a function, then reflect, then generate a list of things to unit tests, then generate function for each unit tests, then use exec to test each unit test"
• The "guidance builder agent" takes this and generate the file I wrote above with the user/assistant and guidance syntax
• At the end of the execution, we get the function and all unit tests in a python list so that we can store it and use it later if we make changes to the function.
  -> Noise reduced

Let me know if we should already discard this idea and keep it simple with OpenAI API calls.

[daveshap]

Seems useful yeah. Constrain the model