Reqnroll.VisualStudio: New project wizard should use latest version of Reqnroll

[clrudolphi]

The new project wizard currently uses a hard-coded version number of the Reqnroll Nuget package that it adds to the project that it's creating. It would be a convenient feature if the wizard instead checked the nuget package feed and referenced the latest version.

[gasparnagy]

Absolutely agree.

Just need to find a simple way to do it. Maybe you can get some ideas on how to do this reliably from this repo (that does much more, but still relatively simple): https://github.com/taori/NugetUnlister

[gasparnagy]

@Code-Grump We update the templates manually, ad-hoc. The problem is that normally you would need to update them after every single patch release. (So every Reqnroll release would trigger a VS extension release.)

@clrudolphi @Code-Grump I think a good compromise could be to just use the latest published version of Reqnroll (that is in our control), but not the other dependencies, like test framework versions. The test framework versions could be updated manually if our dependencies change (that is more rare).

[Code-Grump]

Is there any way for our templates to be more dynamic and look for assets that are more sophisticated than just the NuGet feed? I'm afraid I'm totally oblivious to this aspect of VS integration.

I'm imagining the template fetching a manifest containing whatever version information is useful (and using a pre-baked manifest in the event of no comms), something we could publish as a static asset whenever we wanted to upgrade the templates without requiring a full installer.

[gasparnagy]

Is there any way for our templates to be more dynamic and look for assets that are more sophisticated than just the NuGet feed? I'm afraid I'm totally oblivious to this aspect of VS integration.

@Code-Grump Yes, this is an option. We can do basically anything. It's pure C# code.

The question here is what this source would be. Or even more importantly what information do we need? Because if we need the information on what versions of dependencies we support, that is in the NuGet package anyway.

[clrudolphi]

@gasparnagy Is there much of a risk that, by taking the latest Reqnroll and supported test framework, that we break the VS extension itself? That is, do we release Reqnroll before the extension can support it?

[gasparnagy]

@gasparnagy Is there much of a risk that, by taking the latest Reqnroll and supported test framework, that we break the VS extension itself? That is, do we release Reqnroll before the extension can support it?

There is no significant risk. We (I) test the unreleased Reqnroll versions anyway with the latest VS extension, so if we discover a compatibility problem, we would anyway improve the VS extension first far before the actual Reqnroll release. But this is a very rare situation anyway.

[clrudolphi]

I'm experimenting with NuGet.Protocol and making good progress. The intent would be to parameterize the ProjectTemplate.csproj file within the Reqnroll.VisualStudio.ProjectTemplate project.

General algorithm will be:
Given testFramework and target runtime, find the latest version of "Reqnroll.testFramework"
Fetch the dependencies of that.
Find the versions of the key dependencies that are needed to be mentioned in the ProjectTemplate.csproj

• For Reqnroll.xUnit:
  • xUnit
  • xunit.runner.visualstudio
• For Reqnroll.MSTEST:
  • MSTest.TestFramework
  • MSTest.TestAdapter
• For Reqnroll.NUnit:
  • nunit
  • NUnit3TestAdapter

If fetching these version identifiers dynamically fails, fall back to hard-coded values like we have today.

Q:

1. The version constraints for the test frameworks (within the Reqnroll.xxx nuspec) are given as open ended ranges, such as [3.2.8, )
   Therefore, should we take the explicitly mentioned minimum version within the constraint (such as 3.2.8) or take the latest available that fits within the constraint expression?
2. For the test project template, we need to include <PackageReference>s for Reqnroll, the testFramework, and for the testFramework's test adapter. The test adapaters are NOT explicit dependencies of Reqnroll nor the testFramework.
   For xUnit and MSTest, the version number of the test adapter is (generally) always the same as the test framework (that is, if the version of xUnit is 1.1.1, then the version of xunit.runner.visualstudio is also 1.1.1)
   But for nunit this is not true. The version of nunit and the NUnit3TestAdapter test adapter are versioned separately; so how will we know which one to include?
3. I'll search about, but if you have any quick pointers in how to feed these values into the templating engine I'd appreciate it.

[Code-Grump]

I personally find it uncomfortable to rely on assets hosted in GitHub, because the process of making the change live will happen upon merge, before any other processes can run. It can be good, but it always feels risky to mix deployment models. If we want to use GitHub, I think creating a separate repo would be cleaner.

My personal preference would be to host the files under assets.reqnroll.com and publish to there as a pipeline action, but only @gasparnagy knows the effort and feasibility of such an idea.

[gasparnagy]

I think that would be feasible.

Could we draft what information would be included in this manifest file? Let's try to come up with a concrete example.

[clrudolphi]

I was thinking something along the lines of:

{ 
    "xUnit":{
        "Reqnroll.xUnit": "3.0.0",
        "xunit": "2.9.3",
        "xunit.runner.visualstudio": "3.1.0"
    },
    "NUnit":{
        "Reqnroll.NUnit": "3.0.0",
        "NUnit": "4.3.2",
        "NUnit3TestAdapter": "5.0.0"
    },
    "MsTest":{
        "Reqnroll.MsTest": "3.0.0",
        "MSTest.TestAdapter": "3.9.1",
        "MSTest.TestFramework": "3.9.1"
    }
}

We might get a bit more fancy and use version constraint expressions instead of fixed versions. But this is enough to list the set of dependencies that should be populated into the project template and their versions.

[gasparnagy]

We might get a bit more fancy and use version constraint expressions instead of fixed versions.

I think the fixed versions are better, because we need to put fixed versions to the project file anyway.

Actually with such a descriptor file, we could generate all the test framework dropdown logic based on that, so if there is a new one, like xUnit v3, we would just need to add that to the descriptor file. The question is if we need anything else for such a setting besides the dependencies. Because if we need other metadata (like "description"), we might need to have one additional level in the descriptor, like:

{ 
    "xUnit":{
        "description": "Use xUnit v2 test executor with Reqnroll",
        "dependencies": {
          "Reqnroll.xUnit": "3.0.0",
          "xunit": "2.9.3",
          "xunit.runner.visualstudio": "3.1.0"
        }
    },
    [...]
}

[clrudolphi]

I agree with the idea of using the file to drive the dropdowns in the wizard.

What if instead of having dependency version information, it simply had the PackageReference statements needed for each test framework?

Such as

{ 
    "xUnit":{
        "description": "Use xUnit v2 test executor with Reqnroll",
        "dependencies": {
           "<PackageReference Include=\"Reqnroll.xUnit\" Version=\"2.2.1\" />
           <PackageReference Include=\"xunit\" Version=\"2.8.1\" />
           <PackageReference Include=\"xunit.runner.visualstudio\" Version=\"2.8.1\" />"
        }
    },
    [...]
}

A bit more verbose, but it might simplify the logic when substituting in the text into the template. If the csproj template is NOT the only place where this info is needed, then I would revert back to the structured dependency information as we envisioned earlier.

[clrudolphi]

• Would you want the URL of the descriptor file to be hard-coded into the wizard or part of some configuration file somewhere?
• When should the fetch of the descriptor file happen - at extension launch time or when the NewProject Wizard is run?
• What kind of time-out value would be appropriate? I'm thinking something short in order to keep the launch of the wizard responsive.
• Should the fetch of the descriptor file happen on a background thread? What existing code within the Extension should be leveraged to do that and to communicate results back to the Wizard? (For instance, the IIdeScope has a FireAndForgetOnBAckgroundThread method; would that be appropriate?)
• What caching behavior should the Extension provide of the fetched descriptor? None? (aka, the wizard re-fetches everytime the New Project Wizard is invoked) If anything more sophisticated, can we depend upon the server to provide some kind of cache-header to indicate cache validity timeframe?
• Would you want the monitoring service to include a flag whether the wizard used the values from the retrieved descriptor file or fell-back to the hard-coded values? Would you want other analytics events added regarding fetching the descriptor file?

[clrudolphi]

@gasparnagy Hoping you can point me in the right direction. The current ProjectTemplate has a set of PackageReferences like this:

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.10.0" />$if$ ('$unittestframework$' == 'xUnit')
    <PackageReference Include="Reqnroll.xUnit" Version="2.2.1" />
    <PackageReference Include="xunit" Version="2.8.1" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.8.1" />$endif$$if$ ('$unittestframework$' == 'NUnit')
    <PackageReference Include="Reqnroll.NUnit" Version="2.2.1" />
    <PackageReference Include="nunit" Version="3.14.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="4.5.0" />$endif$$if$ ('$unittestframework$' == 'MSTest')
    <PackageReference Include="Reqnroll.MsTest" Version="2.2.1" />
    <PackageReference Include="MSTest.TestAdapter" Version="3.4.3" />
    <PackageReference Include="MSTest.TestFramework" Version="3.4.3" />$endif$$if$ ('$fluentassertionsincluded$' == 'True')
    <PackageReference Include="FluentAssertions" Version="6.12.0" />$endif$
  </ItemGroup>

The switch between the available choices of test framework is controlled by the $if$ statements. Where will I find documentation on the available set of operators in this templating language? Does it provide a $forEach$ (iteration) construct?

I'm asking because I've tried a method whereby I simply provide a string that includes all of the PackageReference lines for a given test framework as a single substitution variable, but the templating engine (or VS/MSBuild) doesn't like it and drops it, so I end up with a generated project csproj that is missing all of its package references.

[clrudolphi]

we'd likely be able to rely on TLS root trust to know we've reached our trusted host.

We would have to pin the id and public key of the cert for reqnroll.net in order to validate this, right?

I'm assuming any actor savvy enough to pull off a DNS hijack can also manage to obtain a cert for reqnroll.net as well.

Ugh. Just thinking about this makes my head hurt. With the industry moving to very short lived certs, pinning a key value won't be feasible either.

[clrudolphi]

Actually just thought of a simpler solution. The extension will keep a hard coded list of package names and will only configure template substitutions for those packages. If the descriptor file mentions something outside that list then the entire file is ignored and we fall back to the preconfigured set of dependencies and versions.

[Code-Grump]

No need for cert pinning if we use root trust methods. If the certificate is not signed by a trusted authority, we reject it. Normally "trusted authority" means "is in the Windows trusted certificate store" and it's the foundation for all TLS security on Windows. If we can't trust that, we have bigger problems! 😄

I can go into more details on this, but I feel like we're well into "good enough" territory.

[gasparnagy]

Security: I think you are right: the TLS verification provided by the HTTP stack used by HttpClient is probably enough. The prefix check might not be necessary even, but I think we should go for the structured dependency configuration option and not to the "include a project snippet in json" approach, so that there are less options to be afraid of.

Templating: The options are limited (I don't think there is a foreach), but you can just build the entire reference group text from code and pass it in via a single parameter, like

  <ItemGroup>
    $projectreferences$
  </ItemGroup>

[clrudolphi]

A first draft is in PR 93. This has the basic structure that I was thinking of.

Unfortunately, the parameter substitutions into the csproj file aren't working and I'm getting cross-eyed trying to find my mistake.

Take a look and I welcome any feedback. This only represents a few hours of effort, so I'm not tied to anything here. We can change it.

[gasparnagy]

I don't know what the problem could be, the substitution code looks ok. Maybe it does not allow underscores in the placeholder names?

[clrudolphi]

I don't know what the problem could be, the substitution code looks ok. Maybe it does not allow underscores in the placeholder names?

Indeed, it seems that the underscores were the problem.