Test Plan

Project

The following projects can be used (but not limited to):

• Junit4:
  • https://github.com/angelozerr/lsp4xml
  • https://github.com/Microsoft/todo-app-java-on-azure
• Junit5:
  • https://github.com/junit-team/junit5-samples
• TestNG:
  • https://github.com/spring-projects/spring-boot/tree/master/spring-boot-samples/spring-boot-sample-testng
  • http://www.mkyong.com/wp-content/uploads/2014/01/TestNG-Spring-Example.zip

Version

1. Check the version is equal to the current milestone.
2. Check the changelog has the right version and no typo.

Basic

1. The following events will startup the extension:

• Workspace contains pom.xml
• Workspace contains build.gradle
• Test explorer gets focus
• Trigger the extension commands

2. The startup activation time of the extension should no more than 1000ms.

Test Explorer - UI

1. The explorer should perform as lazy-load behavior: only load the children nodes if the parent nodes are expanded.
2. The nodes should all be sorted alphabetically.
3. If the method has annotated with @DisplayName (a JUnit 5 annotation), the node will use the annotation value as it's display name.
4. The hierarchy should be correct - Both for Single-Root or Multi-Root scenarios.
5. Click on the class/method level node, the editor should jump to the correct position of the source file.
6. Update/Delete/Create test code, the explorer should be updated accordingly.
7. The explorer can be refreshed if the user clicks refresh in either navigation bar or context menu.

Test Explorer - Run/Debug Test

1. There are actions called run test/debug tests in both navigation bar or context menu. In other words, the tests can be run/debug in the following level:
   1. Workspace Root
   2. Workspace Folder
   3. Package
   4. Class/Inner Class
   5. Method
2. After executing run tests/debug tests, all the tests under the target node will be run.
3. After executing run tests/debug tests, in the notification window at the right-bottom side, it will show the status of the current test tasks, such as: searching test items..., then running tests...

Code Lens

1. Code Lens should be shown at the following places:
   1. Test Methods (Annotated with @Test).
   2. Classes which contains test methods.
2. By default, the Code Lens Contains following actions:
   1. Run Test
   2. Debug Test
3. After tests are run, a new status icon should be added as the third Code Lens action. For those tests that have not been run, the status icon should not be shown for it.
4. Click Run Test will run the tests
5. Click Debug Test will debug tests
6. Click the status icon should show the Test Report

Test Report

1. Only show the filter button when necessary. For example, the Failed button will appear only when there are failed tests. All of them can filter the results correctly.
2. Test report theme updates when the user changes the color theme.
3. The report can auto-update if the user reruns the tests and get different results.
4. The test report is read-only. (Not prompt saving when closing it).
5. The methods are categorized by classes.
6. Can navigate to the method source from the report.
7. Can config the report position through java.test.report.position

Test Status Bar

1. Should show Running Tests... during running.
2. When the status bar is showing Running Tests..., click it should open the output channel.
3. After running/debugging tests, the status bar should show the result.
4. Click the result, will open the Test Report.

Cancel the running job

1. start a test run that lasts for a long period, for example, add Thread.sleep in your test.
2. click Cancel button on the notification message, the running job should be stopped.

Logs

1. Can trigger command Java: Show Test Output to open the output channel.
2. Can trigger command Java: Open Test Runner Log to show the log file.
3. Can set log level: verbose, info, error

Run With Configuration

1. When having non-empty setting for java.test.config, the extension should honor java.test.config, otherwise, if java.test.config is not set or is empty, the extension will try to read the launch.test.json
2. If launch.test.json is read, the extension will show a message box, hinting that the user is using the deprecated configuration. The user can: More details: View documents or Never show: Do not show this message any more
3. java.test.config could be either an array or an object
4. If java.test.config is not empty, user will see an list to select the configuration
5. After user selects the configuration, the extension will pop up a message box. The user can: Yes: Set the selected config as default, No: Do nothing or Never show: Do not show this message any more
6. The ${workspaceFolder} is available in workdingDirectory, args, vmargs and the value of entries in env