Regarding the Use of a Pure Visual Mode to Drive Workflows

[Explorer1092]

After reading through the source code, I understand that the current workflow is as follows:

• The system captures a screenshot of the page and retrieves element information.
• The screenshot and element information are sent to the LLM.
• The LLM analyzes the screenshot and decides which element to interact with (returns the element index).
• The controller retrieves the coordinates of the element from the known list of elements based on the element index.
• The system uses these coordinates to perform the action.

If the browser cannot return element information, even if the LLM can identify the element to interact with from the screenshot, the system will not be able to translate the LLM's decision (element index) into actual coordinates because:

• state.interactive_elements will be empty.
• When the LLM returns an element index, the click_element method will not be able to find the corresponding element and its coordinates.

Is there any consideration for using a pure visual mode to drive the workflow in the future?

In my previous experiments, I found that the browser's capability to provide element indices is insufficient. It is easy to lose key elements. Therefore, I believe that using a pure visual mode, or making it the primary mode, is a good trend.

Additionally, I would like to know if there is any intention to develop this feature.

[skull8888888]

@Explorer1092 hey there, not really sure what are you trying to say here, primarily confused by this part

Therefore, I believe that using a pure visual mode, or making it the primary mode, is a good trend.

Did you mean that it's not a good trend?

That thing aside, I do believe that vision only approach is robust as LLMs are getting really good at vision understanding. I think it's a right bet, and our element identification is the best on the market, so it all should be fine.

[Explorer1092]

It's a good trend, but when I read the source code, I found that the current attention relies on getting elements from the browser, rather than directly asking the model and getting the coordinates of the target element.