[Feature Request] Code execution mode for models without function calling feature

[dandansamax]

Required prerequisites

• I have searched the Issue Tracker that this hasn't already been reported. (+1 or comment there if it has.)

Motivation

Current framework only support VLM with function calling feature. It limits our supporting models. The function calling feature by generating code. For exmaple, if an model with out function calling want to call click(11) with write_text("enter"), it can generate a code block:

click(11)
write_text("enter")

This mode will support almost all existing VLMs.

Solution

No response

Additional context

No response