Add new release status trait

[nicrowe00]

Adding a new trait to the test runner that reports whether the version of .NET being tested is a released, preview, rc1 or rc2 version of .NET.
Trait appears in the output as 'status='.

To skip tests using this trait, in the tests JSON file enter:
status=released
status=preview
status=rc

[omajid]

Could you please update tests too?

Any particular reason that one status captures all preview releases but two different values are used for rc1 and rc2? Did you consider a ga vs non-ga distinction?

[nicrowe00]

Any particular reason that one status captures all preview releases but two different values are used for rc1 and rc2? Did you consider a ga vs non-ga distinction?

This was mostly due to the naming conventions of the packages, how there's packages labeled preview, packages labled rc1 and packages labled rc2. I don't mind switching to simply just say if the .NET version is GA or non-GA.

[nicrowe00]

There is a benefit to the "preview, rc1, rc2" distinction in that tests can be set skip preview versions while still running on rc1 or rc2 and vice versa but realistically due to the progression of .NET packages I doubt there will ever be a use case for that.

[omajid]

I see. Do you think it makes sense to combine rc1 and rc2 phases into a single rc phase (to mirror preview)?

[nicrowe00]

I see. Do you think it makes sense to combine rc1 and rc2 phases into a single rc phase (to mirror preview)?

Makes sense to me. I've added this change in the latest push.

[tmds]

Currently for an upcoming release, we can either let the test fail, or we can add version=10 as a skipped trait if we want to skip it for that version.

@nicrowe00 why do we need something in addition? can you explain the use-case?

[nicrowe00]

Currently for an upcoming release, we can either let the test fail, or we can add version=10 as a skipped trait if we want to skip it for that version.

@nicrowe00 why do we need something in addition? can you explain the use-case?

@tmds This is for tests that aren't expected to pass while we're in the preview phase but are expected to pass once the new version of .NET is released. Setting it to skip based on this rather than version=10 means the test will automatically start working and passing again after release and there is no need to update the test traits in another PR.

[tmds]

This is for tests that aren't expected to pass while we're in the preview phase but are expected to pass once the new version of .NET is released.

Can you give some examples?

[nicrowe00]

The current example for this is dotnet-monitor-works: redhat-developer/dotnet-regular-tests#380.
There is no version of dotnet monitor available currently for .NET 10 previews and as such we've disabled the test for .NET 10: redhat-developer/dotnet-regular-tests#381.
Presumably when .NET 10 is released there will be a version of .NET monitor available so having it set to skip preview versions only will mean it'll run and pass again once it has a released version.

[tmds]

The current example for this is dotnet-monitor-works: redhat-developer/dotnet-regular-tests#380.
There is no version of dotnet monitor available currently for .NET 10 previews and as such we've disabled the test for .NET 10: redhat-developer/dotnet-regular-tests#381.
Presumably when .NET 10 is released there will be a version of .NET monitor available so having it set to skip preview versions only will mean it'll run and pass again once it has a released version.

We want to validate dotnet monitor works when there is a package available.

We should avoid tests that only run against released versions. When they fail, they block the release until they are manually waived.

I think this trait is enabling something we should avoid.

[nicrowe00]

If the package becomes available for it then yes, obviously we should set it to run again.
And I'm not suggesting the next time we do a release we should leave it set to skip the preview versions if they have the required package.
My point is that right now there is no package for preview versions of .NET 10 and we already set the test to skip .NET 10 so clearly we're not holding out hope for the package soon so we should set it to skip the preview specifically for now and if the package becomes available during preview we'll re-enable it and if not then we'll re-enable for the next preview version.

[tmds]

we already set the test to skip .NET 10

Personally, I don't mind having these tests continue to fail until the underlying issue is resolved.

Have we requested Microsoft to release a preview package?

Since @omajid disabled the test, I expect he'll follow up on when the test needs to be re-enabled.

I think we should avoid tests that run only against released versions. They turn into bad surprises.

[tmds]

dotnet-monitor-works was re-enabled: redhat-developer/dotnet-regular-tests#384.

I'm going to close this PR as I don't think it makes sense to pursue this feature further at this time.