Initial CI/CD

[Flash0ver]

I imagine the CI and CD workflows to be a combination of previous projects, but all in GitHub Actions (no Azure DevOps):

• https://github.com/Flash0ver/F0.CodeAnalysis.Benchmarking/tree/main/.github/workflows
• https://github.com/Flash0ver/F0.Analyzers/tree/main/azure/pipelines

Workflows

• CI workflow
  • on push to main and Pull Requests
  • Job "Restore & Build & Test" (individually, as most important job to diagnose potential issues quicker)
    • matrix all 3 OS
    • optional: matrix Debug and Release configuration
    • Step: Restore
      • explicitly use root nuget.config
      • no caching
    • Step: Build (no restore)
    • Step: Test (no build)
      • fail workflow if any test fails
      • in any case, still complete the other OS/Configuration test run
      • if failed, do not run run code coverage nor mutation testing
  • Job: "Collect Code Coverage"
    • package coverlet.collector already added for both test projects
    • on Linux only
    • Step: restore .NET local tools
      • specify NuGet.config and dotnet-tools.json manifest explicitly
      • no caching
    • Step: Collect Code Coverage
      • Format: Cobertura
      • create and use .runsettings
        • to specify <Format>cobertura</Format> explicitly
        • to allow suppressing/excluding namespaces and files in the future
      • produce assets in ./artifacts/code-coverage
    • Step: Generate Report
      • via dotnet-reportgenerator-globaltool, install as local .NET tool
      • merge all Cobertura results via a wildcard pattern
      • Generate HTML
      • produce assets in ./artifacts/code-coverage
    • Step: Publish HTML Report Artifact
  • Job: Mutation Testing
    • via dotnet-stryker, install as local .NET tool
    • on Linux only
    • Step: restore .NET local tools
      • specify NuGet.config and dotnet-tools.json manifest explicitly
      • no caching
    • Step: run mutation testing and generate HTML report
      • create and use stryker-config.json for future configuration
      • produce assets in ./artifacts/mutation-testing
    • Step: publish HTML Report Artifact
• CD workflow
  • on push to publish
  • on Linux only
  • Step: Run Tests (no coverage, no mutation testing, use implicit restore and build, Release-Configuration only)
    • fail pipeline if any test fails
  • Step: Pack
    • dotnet pack produces "Release" assets per default since .NET 8.0 ... but we could still explicitly set it
    • create NuGet package in ./artifacts/package
  • Step: Publish
    • NuGet Secret Name: NUGET_API_KEY
• both
  • mind the new <ArtifactsPath>$(MSBuildThisFileDirectory)artifacts</ArtifactsPath> output paths
    • create output HTML artifacts within ./artifacts
  • set available "no-telemetry", "no-logo" and "no-first-time-experience" environment variables for the jobs
  • set --nologo option per .NET SDK command, if available
  • use specific OS label, that is equivalent to the current ...-latest (reproducible builds)
  • checkout depth: 1 (history is currently not required)
  • use SDK from global.json explicitly (reproducible builds, should an SDK be dropped from a runner, and also currently the .NET 8.0 SDK is not yet pre-installed in the GitHub-hosted runners)
  • use latest GitHub Actions currently available as stable release
  • consider pinning actions to the commit SHA of the release tag (and add actual version comment)
    • see https://twitter.com/Tyrrrz/status/1712422863859274010
  • arguments passed to CLI arguments and options should be environment variables

Additional Workflows / Jobs

• Scan for packages that have known vulnerabilities
  • dotnet list package --vulnerable --include-transitive
    • See https://devblogs.microsoft.com/nuget/how-to-scan-nuget-packages-for-security-vulnerabilities/
    • Triggered by PR, manually, and schedule automatically

[Gh0stWalk3r]

Do you need some help to setup CI/CD?

[Flash0ver]

@Gh0stWalk3r I could use some help indeed ... as I'm currently focusing on getting "Benchmarking" and "Testing" for Roslyn Source Generators going, that I then also intend to dog-feed this project with.

I updated the description with what I had in mind so far. But different suggestions are welcome!
I know it's a lot ... feel free to build a smaller subset: e.g. skip Mutation Testing for now ... I'll open follow-up issues for the parts not included for now.

Thank you for your interest in contributing to FlashOWare.Generators. I appreciate it!

[Gh0stWalk3r]

Looks very well thought out 👍🏼

Another possibility to publish new packages is to manually create a releases in GitHub or applying a tag, which will run the publish (CD) workflow.
This way you are applying version tags to e.g. the master branch and do not need a publish branch.
Releases could also contain a generated summary of the changes/commits.
When working with releases, do you also want to attach the published NuGet package to the release as artifact/asset?

In the meantime I will start working on the CI workflow.

[Flash0ver]

I love the idea of publishing a package triggered by creating a Release. My publish-branch-"flow" I used so far was only due to my lack of knowledge that new (Pre-)Releases can also be a trigger (I'm quite bad with GitHub Actions, I really appreciate the help here!). I guess we can get rid of the publish branch altogether.
Will that also trigger when editing a release? E.g. adding/editing description, or only on initial creation with a tag?

I don't necessarily require the NuGet package to be part of the artifacts / assets ... but if you think that's a good idea, please feel free to add it. I mean, as part of a Release, the package will be on NuGet.org anyway.
But, from a workflow perspective: perhaps not on every push to a PR, but only on merging a PR to main ... so that we at least also dotnet pack and get potential NuGet Warnings as Errors to fail the build. And maybe not adding the package to the artifacts of a main build, but publishing it to GitHub Packages instead, so that this feed could be used as a "latest canary" feed for testing and experimentation. Would be cool to use the "current" version + the main's commit SHA as SemVer-Style "Build Metadata" for the GitHub Packages version. But I'm a bit unsure how to auto-increment the version number correctly here. Perhaps we do need something like Nerdbank.GitVersioning and can no longer have a Fetch-Depth of 1.
What do you think about that?

(I have the feeling I'm becoming victim of feature creeping ... let's develop a bit of a "plan" here, and then open incremental issues for the CI/CD scope)

[Gh0stWalk3r]

I intend to post the reports in the pull requests as markdown.
@Flash0ver can you please check if the read/write permission for actions/workflows is enabled?
They can be found in Settings > Actions > General > Workflow
Info taken from the plugin creator: https://github.com/marocchino/sticky-pull-request-comment/tree/v2/?tab=readme-ov-file#error-resource-not-accessible-by-integration

[Flash0ver]

✔️ done (workflow permissions)