Test Plan

Project

The following projects can be used (but not limited to):

• Junit4:
  • https://github.com/angelozerr/lsp4xml
  • https://github.com/checkstyle/checkstyle
• Junit5:
  • https://github.com/junit-team/junit5-samples
• TestNG:
  • https://github.com/spring-projects/spring-boot/tree/master/spring-boot-samples/spring-boot-sample-testng
  • http://www.mkyong.com/wp-content/uploads/2014/01/TestNG-Spring-Example.zip

Version

1. Check the version is equal to the current milestone.
2. Check the changelog has the right version and no typo.

Basic

1. Open a Java project folder
2. Open a Java file
3. Check if the extension will be activated.
   1. Explorer view will be shown.
   2. A new output channel Java Test Runner will be created.
4. The startup activation time of the extension should no more than 1000ms.

Test Explorer - UI

1. The explorer should perform as lazy-load behavior: only load the children nodes if the parent nodes are expanded.
2. The nodes should all be sorted alphabetically.
3. If the method has annotated with @DisplayName (a JUnit 5 annotation), the node will use the annotation value as it's display name.
4. The hierarchy should be correct - Both for Single-Root or Multi-Root scenarios.
5. Click on the class/method level node, the editor should jump to the correct position of the source file.
6. Update/Delete/Create test code, the explorer should be updated accordingly.
7. The explorer can be refreshed if the user clicks refresh in either navigation bar or context menu.

Test Explorer - Run/Debug Test

1. There are actions called run test/debug tests in both navigation bar or context menu. In other words, the tests can be run/debug in the following level:
   1. Workspace Root
   2. Workspace Folder
   3. Package
   4. Class/Inner Class
   5. Method
2. After executing run tests/debug tests, all the tests under the target node will be run.
3. After executing run tests/debug tests, in the notification window at the right-bottom side, it will first show searching test items..., then running tests...

Code Lens

1. Code Lens should be shown at the following places:
   1. Test Methods (Annotated with @Test).
   2. Classes which contains test methods.
2. By default, the Code Lens Contains following actions:
   1. Run Test
   2. Debug Test
3. After tests are run, a new status icon should be added as the third Code Lens action. For those tests that have not been run, the status icon should not be shown for it.
4. Click Run Test will run the tests
5. Click Debug Test will debug tests
6. Click the status icon should show the Test Report

Test Report

1. Only show the filter button when necessary. For example, the Failed button will appear only when there are failed tests. All of them can filter the results correctly.
2. Test report theme updates when the user changes the color theme.
3. The report can auto-update if the user reruns the tests and get different results.
4. The test report is read-only. (Not prompt saving when closing it).
5. The methods are categorized by classes.
6. Can config the report position through java.test.report.position

Test Status Bar

1. Should show Running Tests... during running.
2. When the status bar is showing Running Tests..., click it should open the output channel.
3. After running/debugging tests, the status bar should show the result.
4. Click the result, will open the Test Report.

Cancel the running job

1. start a test run that lasts for a long period, for example, add Thread.sleep in your test.
2. click Cancel button on the notification message, the running job should be stopped.

Logs

1. Can trigger command Java: Show Test Output to open the output channel.
2. Can trigger command Java: Open Test Runner Log to show the log file.

Run With Configuration

1. Run tests from test explorer: Run With Configuration. For env variables, check System.getenv() works.
2. Run tests from test explorer: Debug With Configuration.
   1. For env variables, check System.getenv() works.
   2. Check variable evaluation works.
3. Check default configuration works fine
   1. Edit test configuration, fill in default section with the name of the config that you specify as the default configuration.
   2. Click Run Test in Code Lens, verify the default configuration is picked(e.g. set up the working directory in the config)