Move text-only packages from SBRP to SBE

[MichaelSimons]

Currently SBRP has three types of packages: reference, target and text-only. Text-only packages are not reference packages rather they are a type of external package. They are a special type of source-build-externals rather than a SBRP. This is re-enforced with how text-only packages are treated by the poison infrastructure. Reference and target packages are prohibited in the build output while text-only packages are allowed. Because text-only packages are produced from SBRP, they appear in vmr's prereqs/packages/reference folder rather than "regular" packages.

The primary motivation for changing this now is related to including all SBRPs in the PSB archive. The only remaining reason for including SBRP packages in PSB this is because of bootstrapping logic for Microsoft.Build.NoTargets and Microsoft.Build.Traversal (both are text-only packages). If text-only packages were treated like the other SBE packages, SBRP and associated special logic to include them in PSB could be removed.

Related: #4976

[ViktorHofer]

I like this proposal and I absolutely agree on that packages like microsoft.build.notargets and microsoft.build.traversal belong into SBE. Same for wcwidth.sources, microbuild.core, microsoft.docker.sdk or any other tool that the .NET Team doesn't produce. I would say that they belong there as submodules, as the rest of SBE but I think that's an implementation detail and isn't that important to me.

What I struggle with is agreeing on is saying that all text-only packages belong into SBE. Let's take System.Composition as an example (admittedly, that's currently not in SBRP but could be taken a dependency on at any time by product repos). We, the .NET Libraries team own that and produce it live in runtime. We call that a meta package as it doesn't contain any assemblies or content, it only brings additional packages in via package dependencies.

When we add a new version of the System.Composition.* packages (i.e. System.Composition.Hosting) to SBRP it would feel very strange to require sending another PR to SBE to add the System.Composition package with that same version. Logically I see the meta package and its dependencies as strongly connected.
It's also produced by dotnet/runtime so the "Externals" part in SBE's repo name would be misleading.

Other packages that fall into that bucket that exist in SBRP text-only are "microsoft.netcore.platforms", "microsoft.private.intellisense", "microsoft.codeanalysis.collections" and "microsoft.extensions.commandlineutils.sources".

This is re-enforced with how text-only packages are treated by the poison infrastructure. Reference and target packages are prohibited in the build output while text-only packages are allowed.

In the example of System.Composition, the package references multiple TFMs: net462, netstandard2.0, net8.0, net9.0 and net10.0. When the package would get added as a text-only package to SBRP, the net462 TFM would get omitted. That's an indicator to me that the package differs from its origin and should be prohibited in the build output. This is a grey area as it doesn't contain any content so it might not matter that much but if anyone would take this package and publish it to a feed, things would be fundamentally broken (i.e. building System.Composition API targeting .NET Framework on Linux).

---

At the end of the day I don't think that repo boundaries matter much for this discussion (SBE and SBRP could be the same repo and we would still have this conversation). I only see that relevant for the dev UX when adding new packages, which should be easy and not require submitting PRs to multiple repos for the same package "family".

It's the categorization of these packages that matters. Something produced by the .NET team shouldn't be treated as external. Packages that reference each other should live next to each other (System.Composition -> System.Composition.Hosting). Packages or package content that are different than their origin should be poisoned to never get published or redistributed.

I hope my feedback makes sense. Happy to chat more about this online/offline.

[mthalman]

I think metapackages are their own thing and shouldn't be treated as text-only packages. In other words, a text-only package has both no code and no package dependencies. Metapackages aren't really handled by SBRP right now.

[ViktorHofer]

Meta packages were part of text-only SBRP in the 5.0 days: https://github.com/dotnet/source-build-reference-packages/tree/release/5.0/src/textOnlyPackages/src/system.composition/1.3.0

That package then later got removed when repos switched away from it.

---

Support for supporting meta packages as text-only was explicitly preserved when we reworked the generate functionality in SBRP. Here's an example that shows that they still work:

C:\git\source-build-reference-packages>generate.cmd -p System.Composition,9.0.0 -x
Restore complete (0.8s)
  PackageSourceGeneratorTask succeeded (0.8s) → artifacts\bin\PackageSourceGeneratorTask\Debug\net10.0\PackageSourceGeneratorTask.dll
  PackageSourceGenerator succeeded (1.1s) → Package source generation skipped for System.Composition (v9.0.0) as it doesn't contain any compile items.

Build succeeded in 5.3s

C:\git\source-build-reference-packages>generate.cmd -p System.Composition,9.0.0 -x -type text
Restore complete (0.7s)
  PackageSourceGeneratorTask succeeded (0.6s) → artifacts\bin\PackageSourceGeneratorTask\Debug\net10.0\PackageSourceGeneratorTask.dll
  PackageSourceGenerator succeeded (1.4s) → Package source successfully generated for System.Composition (v9.0.0).

Build succeeded in 5.1s

C:\git\source-build-reference-packages>git status
On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        src/textOnlyPackages/src/system.composition/

[mthalman]

Yeah, but does it actually build?

[ViktorHofer]

Michael and I had a discussion about this earlier today but FWIW generate.cmd -p System.Composition,9.0.0 -x -type text produces the correct sources for the meta package and adds P2Ps to the referenceOnly project files.

[MichaelSimons]

After discussing w/Viktor I am closing this in favor of #5193 and #5194