Is ICommandLineOptions.IsOptionSet("server") still the right way to detect Test Explorer in .NET 10?

[bradwilson]

I know that today (pre-10), Test Explorer ends up passing --server to the command line, whereas dotnet test ends up passing --internal-msbuild-node. Based on a suggestion from this team, we use the presence of --server as the indicator that we're running in Test Explorer:

https://github.com/xunit/xunit/blob/6b5dc41c1933c85f370d4a991cee4406c5f877f6/src/xunit.v3.runner.inproc.console/TestingPlatform/TestPlatformTestFramework.cs#L264-L265

Given the new dotnet test implementation coming in .NET 10, I wanted to make sure that my existing code which is detecting "server mode" (aka, Test Explorer) is still correct.

[nohwnd]

Yes, test explorer will remain using --server. There are 2 different protocols used to communicate. in dotnet test we use binary messages thought pipe, in test explorer we use the json rpc protocol.

The spec for the pipe protocol should be coming soon. #6187

[bradwilson]

So my check for Test Explorer remains valid?

[nohwnd]

Yes, TE will continue using --server, and it will continue using the RPC protocol.

[Evangelink]

I don't think this is a safe way to check for TE. In the future anything could be using --server, we could also update our implementation to be using the pipe solution.

This is working for now but that's leaking internal implementation details.

I agree that there is no short plan to change anything but that's still not a reliable solution.

[Youssef1313]

Test Explorer uses Json RPC protocol as of today.
The dotnet test coming in .NET 10 will use pipe protocol, but it will not be using the --internal-msbuild-node. It will be using --server dotnettestcli.

I haven't checked what exactly you do when you detect you are running in server mode, but if you are looking specifically for Json RPC, then:

testfx/src/Platform/Microsoft.Testing.Platform/Hosts/TestHostBuilder.cs

Lines 194 to 195 in 0d5aa69

	bool hasServerFlag = commandLineHandler.TryGetOptionArgumentList(PlatformCommandLineProvider.ServerOptionKey, out string[]? protocolName);
	bool isJsonRpcProtocol = protocolName is null || protocolName.Length == 0 || protocolName[0].Equals(PlatformCommandLineProvider.JsonRpcProtocolName, StringComparison.OrdinalIgnoreCase);

[bradwilson]

To be clear, I don't care about the protocol. I only care to know when I'm running in Test Explorer vs. dotnet test. In the past the team suggested checking for the server command line option, but that sounds insufficient based on what you're saying for .NET SDK 10.

What I'm looking for is the official guidance on detecting when we are running inside Test Explorer vs dotnet test. Nothing else matters to me.

[Youssef1313]

The only way I see today is to rely on the fact that Json RPC is used. But this, IMO, is an implementation detail of how Test Explorer works today.

If Test Explorer decided to use the pipe protocol in future, then we will need to somehow expose that piece of information so that you can tell whether it's IDE or command-line.

Alternatively, I think it might be best to actually have a separate test node state for "explicit" tests (which I think is the scenario you need this information for). Then you just don't care about who is running the test app. You simply always report that it's skipped because it's explicit, then Test Explorer and dotnet test are responsible to understand how to deal with that information (e.g, dotnet test can decide to treat both skipped and explicitly skipped the same, while Test Explorer can decide to ignore the explicitly skipped completely, or even show a different visual indicator for it). But, that needs to be implemented on our side first :)

The best you can do today is:

• Follow

  testfx/src/Platform/Microsoft.Testing.Platform/Hosts/TestHostBuilder.cs

  Lines 194 to 195 in 0d5aa69

  	bool hasServerFlag = commandLineHandler.TryGetOptionArgumentList(PlatformCommandLineProvider.ServerOptionKey, out string[]? protocolName);
  	bool isJsonRpcProtocol = protocolName is null || protocolName.Length == 0 || protocolName[0].Equals(PlatformCommandLineProvider.JsonRpcProtocolName, StringComparison.OrdinalIgnoreCase);

  (but again, IMO it's implementation detail of Test Explorer)
• Open a feature request (if not open already) for a new test node state for explicit tests. Or alternatively, it can be just a TestMetadataProperty, like IsExplicitTest which will be well-recognized by Test Explorer.

[bradwilson]

If Test Explorer decided to use the pipe protocol in future, then we will need to somehow expose that piece of information so that you can tell whether it's IDE or command-line.

This is really what I want: a supported, documented way to do this, so that we aren't coming back to a future Visual Studio or .NET SDK release and saying "okay, now what?".

I will open a new issue.

[bradwilson]

#6494

[bradwilson]

As mentioned in the issue, Explicit tests are only half the story (and yes, I've had an issue open for that since March 2024, though of course the most important part of that would be Test Explorer understanding it). The other half is optimization of theory data pre-enumeration; we want it for Test Explorer (and any interactive UI), but not for dotnet test (where it's wasted cycles and memory during discovery).