List of exit codes for vstest.console.exe and dotnet test using MSTest v3

[Cjewett]

Hi all,

I'm trying to figure out if exit code 1 in vstest.console.exe and dotnet test means all tests were executed but not all tests succeeded. We're trying to mark that case as successful in a CI pipeline because the test processing will handle if it was a failure or not. However, if dotnet test fails because of a crash or something more sinister than a failed test I'm wondering if that has a different exit code. Possible there is a list of exit codes and the associated reasoning for vstest / dotnet test?

Apologies if I'm asking this in the wrong repo. Can move it to the vstest repo if needed, but I've had good, responsive discussions in this repo in the past so figured I'd start here first.

Thanks

[Evangelink]

Hi @Cjewett,

With VSTest (under dotnet test) you will have only 2 exit code: 0 when ok and 1 when something is wrong. But please do not a lot of "default" strictness you would expect is not present with VSTest.

Have you tried updating to MTP?

[Cjewett]

Hey @Evangelink, thanks for the response as always!

I took a look at MTP and the docs around exit codes describes exactly what we want but it's a bit of an effort for us to get from here to there. If that's the long-term answer though then I can surface that and get it prioritized.

[Evangelink]

I would be interested to understand why you consider this as a big effort? This is important for us to understand what kind of info or tools we would need to provide to help our users and customers migrate off of VSTest.

If you are already on relatively new versions of MSTest, that's "basically" just adding a Directory.Build.props at the root (or test folder) and updating the build scripts.

[adc-cjewett]

It's mostly a tech debt problem. We've got a bunch of custom scripts working outside of the MSBuild ecosystem and calling vstest.console.exe manually that is doing some nasty stuff.

Luckily we've been refactoring that for a bit and are just about at the last step, but as you can imagine it's taken a bit of time to take our CI pipeline and put it back on the tracks so to speak.

Some examples of issues we've faced:

• Our CI pipeline would manually copy bin directories to parallelize test runs instead of rely on DeploymentEnabled. We ran into issues where files we expect to be copied aren't being copied. In addition to that we were also running into some odd binding redirect issues which we're hopefully at the finish line on now, but takes time to resolve those.
• At some point not all of our test projects were compatible with the dotnet test CLI which we had to resolve.
• Upgrading to the most recent MSTest versions resulted in CI pipeline breakages due to dependency versioning hell and things going haywire when the packages aren't set to the same version, are being downgraded, etc.
• We heavily rely on the TestId's from MSTest to support our quarantining test infrastructure, so any version update there requires us to heavily scrutinize what's going on.

We'll probably be at the point in a month or two where we can "just use Microsoft.Testing.Platform" and moving forward making any changes will be very, very simple for us.

[Evangelink]

@Cjewett are you fine with closing this question?