Main bugs bothering us at the moment

[ffMathy]

I am filing this as a discussion, as I don't believe the current repo structure is very friendly towards examples where a repro can't be given. This is one of those cases. But I still think you guys deserve to know of the instability in the extension that has been bothering us since the rewrite of the extension.

You guys are doing a phenomenal job (that's also why I am sponsoring), and it's amazing to see almost daily releases of the extension sometimes. So this is not meant as a rant. It's meant to be constructive.

Here are the main issues. Again, I'd love to provide a repro, but haven't been able to for the past many months. Either due to time pressure (Q4 goals coming up), or a complex setup causing it to take too many company resources to do so.

I know that you're thinking "well, put your money where your mouth is", and companies are generally not very understanding of contributing to open source. That sucks. As an open source contributor myself, I understand this. However, you may still have interest in fixing this, since I am assuming it affects a lot of other users too.

Timer not resetting

We often get in a situation where the timer just keeps running. This happens typically if we press "refresh" while a test is running, or if we click "stop" while a test is running.

It makes me wonder if finally clauses are being used correctly to properly dispose resources in the event of failures or other scenarios?

Stop has to be pressed twice

Often, when running a long-running test, pressing stop once isn't enough. At the second press of "stop", the test truly stops.

Race conditions while tests are being discovered

While tests are being discovered, if you try to run a test, the whole extension goes into a limbo where you no longer can run anything until you've refreshed all tests again. So we've developed a habbit of waiting a lot of seconds for the discovery to be done.

Running a single test within a test file marks all other tests as "skipped" in the UI

This is annoying, because often you want to still be able to see if the other tests passed or failed the last time.

Sometimes Vitest just gets in a weird state

This typically happens if I have just been debugging a test and then click "stop" in the debugger. After this, I can't start tests anymore normally (only in debugging) and have to click "refresh".

[sheremet-va]

Often, when running a long-running test, pressing stop once isn't enough. At the second press of "stop", the test truly stops.

This is intentional. Vitest awaits when the test gracefully completes at first. If you have a long running test, it will wait until it's completed (in case you need to do cleanup) - this is also how the CLI works

Running a single test within a test file marks all other tests as "skipped" in the UI

This can be fixed no problem

While tests are being discovered, if you try to run a test, the whole extension goes into a limbo where you no longer can run anything until you've refreshed all tests again. So we've developed a habbit of waiting a lot of seconds for the discovery to be done.

Interesting. This should be possible to fix without issues

[sheremet-va]

I get these errors now

Do you use vitest.globals?

[ffMathy]

Yup, I do 🙂

Also, in regards to timings, I just encountered another bug. If you have a worker count higher than 1 and run all your tests, the timings are actually calculated wrong.

[sheremet-va]

The extension doesn't calculate overall time. VSCode does it by itself by summing all durations probably. We just report how long a single test took to run.

[sheremet-va]

Pushed 1.8.1. It should fix the issue with tests unintentionally marked as skip, and it should also fix an error with "No test suite found in the file" when vitest.experimentalStaticAstCollect is enabled. There is still a known issue with how .each tests are reported, I am trying to find a nice solution there

[ffMathy]

It works. And it's lightning fast. Thanks a lot. We don't have a lot of .each tests yet, but we do have a few. I'll keep this enabled either way, and eagerly await a fix. Nice work.