Automated Testing Framework for GUI Tests

[JeremyYao]

Description

Much of the integration/end-to-end (e2e) GUI testing for Daffodil-VSCode is done manually as described in https://github.com/apache/daffodil-vscode/tree/main/src/tests. Having an automated testing suite for GUI integration/e2e tests would greatly enhance the efforts' testing capabilities and opens up bandwidth for developers significantly.

Currently, we're utilizing @vscode/test-electron for some of our automated tests, but they don't test the GUI in an extensive manner that resembles our manual tests. Additionally, @vscode/test-electronseems to provide limited testing capabilities and a total lack of support for testing webviews.

GUI Testing Frameworks

There seems to be two frameworks for testing GUI-aspects of Visual Studio Code extensions

1. vscode-extension-tester
2. WebDriver.io VSCode Extension Testing Service

vscode-extension-tester utilizes Selenium WebDriver where WebDriver.io VSCode Extension Testing Service uses WebdriverIO. It should be noted that WebDriver.io VSCode Extension Testing Service service was inspired by vscode-extension-tester.

Web Driver Differences

Resources that describe the differences between WebDriverIO and Selenium are listed below

• https://gorillalogic.com/blog-and-resources/using-webdriverio-vs-selenium-webdriver-how-when-and-why
• https://contextqa.com/selenium-vs-webdriverio/
• https://www.wetest.net/blog/webdriverio-vs-selenium-comparison-and-analysis-of-differences-691.html

Summary: WebDriver.io seems to be more modern, developer-friendly. Java-script focused. Flexible. More abstracted, and gets tests up and running more quicker. Selenium has been around longer, is more established.

Examples

vscode-extension-tester example

https://github.com/redhat-developer/vscode-extension-tester-example

Ran through examples on my own end and got it to work.

WebDriver.IO VSCode Example

Note: Unable to produce an example. Running through npm create wdio@latest ./, using the following options

PS C:\Users\jeremy.yao\repos\wdio_test> npm run wdio

> wdio
> wdio run ./wdio.conf.ts

2025-03-20T18:36:42.368Z ERROR @wdio/config:ConfigParser: Failed loading configuration file: file:///C:/Users/jeremy.yao/repos/wdio_test/wdio.conf.ts: __dirname is not defined in ES module scope
ReferenceError: __dirname is not defined in ES module scope
    at <anonymous> (C:\Users\jeremy.yao\repos\wdio_test\wdio.conf.ts:59:28)
    at ModuleJob.run (node:internal/modules/esm/module_job:271:25)
    at async onImport.tracePromise.__proto__ (node:internal/modules/esm/loader:578:26)
    at async _ConfigParser.addConfigFile (file:///C:/Users/jeremy.yao/repos/wdio_test/node_modules/@wdio/config/build/node/index.js:267:30)
    at async _ConfigParser.initialize (file:///C:/Users/jeremy.yao/repos/wdio_test/node_modules/@wdio/config/build/node/index.js:239:7)
    at async Launcher.initialize (file:///C:/Users/jeremy.yao/repos/wdio_test/node_modules/@wdio/cli/build/index.js:2263:5)
    at async Launcher.run (file:///C:/Users/jeremy.yao/repos/wdio_test/node_modules/@wdio/cli/build/index.js:2187:5)
PS C:\Users\jeremy.yao\repos\wdio_test> 

Attempted looking on StackOverflow for solutions, implemented some, but to no avail.

My Thoughts

The docs for vscode-extension-tester is more extensive and intuitive to follow upon first glance compared to WebDriverIO's.

I believe it would be worthwhile to look into how integrating each of these two VSCode e2e testing frameworks goes. Simple tasks like can we load the extension in and do something like

• CTRL + Shift + P
• Type in >Daffodil...
• See if an option shows up
• etc...

Discussion

Does anyone have a preference for one of the GUI Testing Frameworks? Any thoughts? Relevant information?

[krishnaharshap]

Hi Jeremy and Team,

Great question! It’s awesome to see the team moving toward more automated testing for the Daffodil-VSCode extension! I wanted to share some thoughts that might help.

My Experience
I’ve used both vscode-extension-tester (which is built on Selenium) and the WebDriverIO VSCode Extension Testing Service for testing VSCode extensions. Here’s how they stacked up for me:

vscode-extension-tester
Pros:
It just works well with VSCode out of the box, and there’s a lot of documentation and real-world examples to follow (as you suggested based on the documentation available at RedHat example repo).
It’s been around for a while, so it’s pretty stable and reliable, especially if you’re running tests in CI/CD.
Doing things like opening the command palette and checking if your extension’s commands show up is pretty straightforward.
Cons:
It can feel a bit “heavy”- setting up the tests takes a little more time, and test runs are a bit slower because of Selenium's architecture.

WebDriverIO VSCode Extension Testing Service
Pros:
It’s more modern and works really nicely if you’re already using JavaScript or TypeScript.
The code is often shorter and easier to read, especially if you like using async/await.
It’s faster to get started if your team is used to web testing.
Cons:
The documentation isn’t as deep, and I ran into some confusing issues when setting it up—especially with things like ES modules and config files (that _dirname issue is a known one; sometimes using import.meta.url or switching to a different config format can help).

My Recommendations:
For your project, I’d probably start with vscode-extension-tester:
It’s more established, and you’re more likely to find answers when you get stuck.
Since it’s already used by a lot of other VSCode extension teams, it’s less risky for long-term maintenance.
Once you have the basics working, you could try out WebDriverIO on the side and see if it offers a better experience for your team.

Tips from my Experience!
Webview Testing: Both tools are still a bit limited here. Sometimes, using something like Puppeteer alongside your main tests helps with webviews.
Avoiding Flaky Tests: Use waits (not just sleeps), and try to keep your tests focused on one thing at a time.
Running in CI: Both tools can run in headless mode (no window needed), but make sure to set up your CI runner to support GUI stuff (like using Xvfb in GitHub Actions).

Thanks for starting this discussion!