How could we expand this to create human language QA automation scripts?

[stefangrotz]

This is a very interesting project. I wonder if there is a easy way to create human language QA scripts that are still reliable. What are your thoughts?

I will experiment with it and document the results here.

[stefangrotz]

Ideas I am testing today:

• Basic smoke test (clicking through all pages of a website and search for obvious errors)
• Use the output to write test reports.

More advanced stuff for later:

• Optical accessibility testing (does with website work for a color blind person?)
• In general: more analysis during browsing
• Convert user stories into test plans and execute them
• compare the behavior between different versions of a website

[stefangrotz]

simple clicking through all pages doesn't seem to work right now. I will try a simpler website next.

I selected the German Wikipedia because it still uses the old design and the menu doesn't has to be opened with the burger icon, which didn't work well in previous attempts.

EDIT: it seems to be a prompting issue. "one after another" doesn't work, but If I explicitly say click the first link on the left menu and then the second, then it works most of the time

EDIT2: I am on the end of my 100 request rate limit for vision for today, but this was fun. I believe if you add an analysis field to the JSON response this could become a useful tool for QA for some use cases

[ishan0102]

This is really interesting, thanks for working on this! I think QA automation is possible but somewhat challenging, as you said the prompt design is really important and GPT-4V is finnicky. I think it's possible that passing the accessibility tree could make this easier to solve, but not sure.

[stefangrotz]

Small update: I think this has been solved much better already by the self-operating-computer project.