Simpler tool usage for smaller LLMs idea

[joaomdmoura]

From discord

Just wanna share an interesting idea regarding more reliable function calling for smaller LLMs without using the native function calling of an LLM provider. This is a replacement for Joao's simple pipe-separated argument string - what I call now lean function calling.

The basic idea, that I got from a Twitter thread, is that most smaller models had been trained on HTML, so you can expect that they can generate HTML reliably. So I experimented using XML as a format for our lean function calling, since it is similar to HTML but allows using arguments as tags. The results are quite promising so far. Try it yourself and feed this prompt into your preferred small model. All 7B models I tried did it correct:

Get the weather for Berlin on 28th of January 2024. And use the weather-tool for it.

The weather-tool function call has the following XML syntax that must be strictly obeyed.

$location
<date_time format="ISO8601>$date_time</date_time>

Mistral-7B-Instruct response example:

Berlin, Germany
<date_time format="ISO8601">2024-01-28</date_time>

Nice, eh. Note the date-time format conversion the model did, because the tag's format parameter is set to ISO8601. This can be used as type hints.

I hacked a simple implementation for my review-tool and I'm testing it. It's working well so far but an issue that still remains, is that sometimes the model does not use all parameters. I think this requires either some prompt engineering or I could add a required parameter to the argument tags. I will share my implementation later.