Improve the UX of api-diff suppression baseline and test resiliency #5356
Description
With dotnet/dotnet#2542 we have enabled api-diffs between Microsoft and source-build SDKs. Resulting diffs are captured in #5344
The implementation uses a built-in feature of ApiCompat task to generate a new suppression baseline, which is further compared with the old baseline. The issue exists with UX of the baseline file. It is not clear which assembly is missing the API, so in the following baseline entry example either left or right assembly are missing the API:
<Suppression>
<DiagnosticId>CP0001</DiagnosticId>
<Target>T:System.Diagnostics.CodeAnalysis.ExperimentalAttribute</Target>
<Left>/tmp/ak2gfs0d.ycb/msft/sdk/10.0.100-rc.2.25465.105/DotnetTools/dotnet-watch/10.0.100-rc.2.25465.105/tools/net10.0/any/BuildHost-netcore/Microsoft.CodeAnalysis.Workspaces.MSBuild.BuildHost.dll</Left>
<Right>/tmp/ak2gfs0d.ycb/sb/sdk/10.0.100-rc.2.25465.105/DotnetTools/dotnet-watch/10.0.100-rc.2.25465.105/tools/net10.0/any/BuildHost-netcore/Microsoft.CodeAnalysis.Workspaces.MSBuild.BuildHost.dll</Right>
</Suppression>
Unlike suppression file, the task's output is specific in calling out the actual assembly missing the type:
error CP0001: Type 'System.Diagnostics.CodeAnalysis.ExperimentalAttribute' exists on /tmp/zezs3i1y.dqa/sb/sdk/10.0.100-rc.2.25465.105/DotnetTools/dotnet-watch/10.0.100-rc.2.25465.105/tools/net10.0/any/BuildHost-netcore/Microsoft.CodeAnalysis.Workspaces.MSBuild.BuildHost.dll but not on /tmp/zezs3i1y.dqa/msft/sdk/10.0.100-rc.2.25465.105/DotnetTools/dotnet-watch/10.0.100-rc.2.25465.105/tools/net10.0/any/BuildHost-netcore/Microsoft.CodeAnalysis.Workspaces.MSBuild.BuildHost.dll
Additionally, the current model would not capture potential bugs, where a specific API, in a single change, disappears on one binary and appears on the other. This can happen if some APIs are conditioned-out on source-build and someone accidentally alters this condition by simply reversing it. The API would disappear from MSFT build, appear in source-build, but the suppression file would not have any differences, thus the test would not fail.