|
| 1 | +# Engineering plan for including .NET Core in Linux Distro Repositories # |
| 2 | + |
| 3 | +## Requirements ## |
| 4 | + |
| 5 | +Requirements for getting .NET Core into other Linux Distros are included in *.NET Core in Linux Distro Repositories*. |
| 6 | +See [Requirements Documents](RequirementsDocument) |
| 7 | + |
| 8 | +[RequirementsDocument]: https://github.com/dotnet/source-build/issues/782 |
| 9 | + |
| 10 | +### Distro Packaging Guidelines ### |
| 11 | + |
| 12 | +Each distro has specific packaging guidelines along with a well-defined review process. These guidelines are similar across distros, with one of the main guidelines being that the source provided contain no pre-built binaries. This is the biggest obstacle to getting .NET Core into distro archives. .NET Core is included in the Red Hat distro today even though the source-build process requires a large number of pre-builts in order to successfully build. Red Hat has given exceptions for us to be able to be included in their distro. |
| 13 | + |
| 14 | +### Toolset for Building .NET Core ### |
| 15 | +There is one exception to the *"no prebuilt binaries rule"* for the case of bootstrapping a toolset. This is an exception that .NET Core will need to take advantage of. This bootstrap process is illustrated below. An existing toolset is used to build a toolset from source. The existing toolset is then thrown away and the newly built toolset is used for the actual build of the assemblies that go into the archive. The source is the same for each stage of the build. |
| 16 | + |
| 17 | +This is discussed in the Fedora packaging guidelines as an exception to the guideline of *no inclusion of pre-built binaries or libraries*. See [Bootstrap Exception](https://fedoraproject.org/wiki/Packaging:Guidelines#Exceptions) |
| 18 | + |
| 19 | + |
| 20 | + |
| 21 | + |
| 22 | +>>Toolset bootstrapping is different than runtime bootstapping provided in [scripts/bootstrap/buildbootstrapcli.sh](https://github.com/dotnet/source-build/blob/92546b3a4000d58b3fb75771fd633f1bef1cfbc7/scripts/bootstrap/buildbootstrapcli.sh). This script is only needed when starting with a new distro on which .NET Core has never built. For the distros being discussed in this document, there are already existing .NET Core runtime and SDK builds available. For that reason, runtime bootstrapping is outside the scope of this document. |
| 23 | +
|
| 24 | +### Runtime vs. SDK ### |
| 25 | + |
| 26 | +In the [requirements document](RequirementsDocument), it is suggested that we *Reduce the problem space and get in the door. Deliver the runtime first.* The full toolset is required to build both the runtime and the SDK. Delivering runtime first doesn't necessarily reduce the problem space because of this requirement to create a bootstrap toolset, first. |
| 27 | + |
| 28 | +In either case, it makes sense to split out the creation of a separate archive package for runtime and toolset with the appropriate package dependency. |
| 29 | + |
| 30 | +### N-1 Toolset ### |
| 31 | +Subsequent versions of .NET Core, when being built for an already bootstrapped distro do not need to follow the bootstrap process as described above. Subsequent versions will use the N-1 version of the toolset from the archive to build. Note, N-1 in this context means the previous version of .NET Core that was included in the distro archive. If, during development of a new version, it is necessary to use a feature from that version, there are two options with regards to source-build: |
| 32 | +1. Ship an intermediate release to establish the toolset baseline. |
| 33 | +2. Re-execute the bootstrap process for the distro to establish a new baseline for the toolset. |
| 34 | + |
| 35 | + |
| 36 | +For patch releases and minor releases, we should be able to stick to building with an N-1 version, since the scale of changes are less and releases are more frequent. When doing major releases, it may be required to use features that are developed in that release because major releases are generally longer and have more new features. As mentioned above, the solution to this is more frequent, intermediate releases. If this isn't possible, the bootstrap process will need to be re-executed. |
| 37 | + |
| 38 | +## Roadmap for 2.1 Support ## |
| 39 | + |
| 40 | +In the document above, the desire is to select a pilot distro to work through the details of submitting to a distro archive. |
| 41 | + |
| 42 | +From the document: |
| 43 | + |
| 44 | +>* _Reduce the problem space and get in the door. Deliver the runtime first._ |
| 45 | +>* _Reduce pre-built binaries to only those required to bootstrap the build_ |
| 46 | +>* _Select a pilot distro from among 'Golden' distros for which we will serve as the primary maintainers_ |
| 47 | +> * _Fedora_ |
| 48 | +> * _Ubuntu_ |
| 49 | +> * _CentOS_ |
| 50 | +> * _Debian_ |
| 51 | +>* _Update source-build to create rpm|deb assets for submission into distro archive acceptance process_ |
| 52 | +>* _Work through the first distro submission process_ |
| 53 | +>* _Add distro archive maintenance to the .NET Core servicing and release processes_ |
| 54 | +
|
| 55 | +One of the 'Golden' distros, Fedora, has an existing special interest group (SIG) that maintains spec files for building .NET Core RPMs. Their goal is to make these the official .NET Core packages in Fedora, but the source does not meet the Fedora packaging guidelines. We should target Fedora as the first 'Golden' distro so we can focus on getting the source to a state that meets the packaging guidelines while the DotNet SIG can continue to focus on packaging for Fedora. https://fedoraproject.org/wiki/SIGs/DotNet |
| 56 | + |
| 57 | + |
| 58 | +## Work Breakdown ## |
| 59 | +With this partnership, there are four main area of focus to get .NET Core ready for inclusion into Fedora's package archive: |
| 60 | +1. Prove out and define process for toolset bootstrapping. |
| 61 | +(https://github.com/dotnet/source-build/issues/724) |
| 62 | +2. Split runtime build and SDK build and define dependencies correctly. (https://github.com/dotnet/source-build/issues/725) |
| 63 | +3. Eliminate prebuilt binaries. (https://github.com/dotnet/source-build/issues/753) |
| 64 | +4. Work with DotNet SIG to get packaging ready. |
| 65 | + |
| 66 | +As stated in the requirements, work will be done in the `release/2.1` branch of source-build. Any changes required to source repos will follow this process: |
| 67 | +1. Include changes in source-build as patches to continue moving work forward. |
| 68 | +2. Write up issue in target repo to include change in `release/2.1` branch and port to `release/2.2` and `master` branches. |
| 69 | +3. Update source-build to move to updated code and remove patch. |
| 70 | + |
| 71 | +Source-build changes will be ported to `release/2.2` and `master` on a regular basis to keep those branches up to date. The desire is to have `release/2.2` and `master` updated and ready to be included in other distros when they are ready to be released. |
| 72 | + |
| 73 | +### Toolset Bootstrapping ## |
| 74 | +This work requires setting up scripts to build the .NET Core toolset from source and then using the result to build source-build. This is to define the process and to ensure that it works consistently by adding CI. It also will require working out issues that we've encountered building source-build with the same version that is being built. See https://github.com/dotnet/source-build/issues/606. This will also track work with source-building [dotnet/BuildTools](https://github.com/dotnet/buildtools) and the eventual move to [dotnet/Arcade](https://github.com/dotnet/arcade), if it is ported to the `release/2.1` branch for any repos. |
| 75 | + |
| 76 | +### Runtime / SDK Split ### |
| 77 | +The main part of this work is to be able to build runtime and sdk and to provide separate RPMs for each, working out the correct dependencies. See this issue: https://github.com/dotnet/source-build/issues/725 |
| 78 | + |
| 79 | +### Prebuilt Binary Elimination ### |
| 80 | + |
| 81 | +There are 6 categories of prebuilts to eliminate: |
| 82 | +1. **Prebuilts that end up in the built product.** |
| 83 | + - These must be eliminated and a process must be put into place to detect and track these to make sure they don't happen again. |
| 84 | +2. **Prebuilts loaded in unneccessary functionality** |
| 85 | + - These are prebuilts that are loaded by a repo's source even though the functionality for the source is not included in source-build. The main group of these are test assemblies, since source-build does not build tests. |
| 86 | +3. **Prebuilts that have a version built with source-build** |
| 87 | + - The source-build process should be modified to get repos to use the source-built version of the package. |
| 88 | +4. **Prebuilts containing only reference assemblies** |
| 89 | + - These prebuilts can be delivered as IL-only code and re-compiled as reference assemblies. |
| 90 | +5. **First party prebuilts** |
| 91 | + - If there are dependencies on 1st party packages that are not built as part of source-build, we should investigate including them in source-build if possible or eliminating the dependency. |
| 92 | +6. **Third party prebuilts** |
| 93 | + - Steps should be taken to remove any dependencies on 3rd party packages. |
| 94 | + |
| 95 | +The first steps to eliminating prebuilt binaries it to setup tools that will track changes to prebuilts within the source-build. The purpose is two-fold: First, it identifies the work remaining and progress that is being made. Second, it catches any regressions in cases of additional prebuilts or prebuilts that get into the built output. The tools should: |
| 96 | +* Track deltas between subsequent builds with regards to prebuilts and their versions. |
| 97 | +* Identify cases where prebuilts end up in the built output. (#1) |
| 98 | +* Be visible and accessible to source-build team members, partners and management. |
| 99 | +* Identify the category of prebuilt, if possible. |
| 100 | + |
| 101 | +### Coordinate with DotNet SIG ### |
| 102 | + |
| 103 | +The DotNet SIG is an existing group that is conducting regular weekly meetings. The source-build team will join and attend these meetings to share and track our plans with the community. |
| 104 | + |
| 105 | +## Addendum: Prebuilt Analysis ## |
| 106 | + |
| 107 | +Analysis is done on latest build of release 2.1 branch on 8/22: |
| 108 | + |
| 109 | +Category | Count |
| 110 | +--- | ---: |
| 111 | +Total Distinct Packages/Versions | 667 |
| 112 | +Total Distinct Package Names | 338 |
| 113 | +Packages that have source-built version | 38 |
| 114 | +Distinct xunit packages | 30 |
| 115 | +Packages that contain only reference assemblies | 188 |
| 116 | +Package usages across all projects | 9593 |
| 117 | + |
| 118 | +### Distinct Count of Package/Versions by Project: ### |
| 119 | +Project | Distinct Package/Version Count |
| 120 | +--- | ---: |
| 121 | +unknown | 205 |
| 122 | +/src/cli | 233 |
| 123 | +/src/clicommandlineparser | 63 |
| 124 | +/src/cli-migrate | 113 |
| 125 | +/src/common | 93 |
| 126 | +/src/coreclr | 52 |
| 127 | +/src/corefx | 177 |
| 128 | +/src/core-setup | 136 |
| 129 | +/src/fsharp | 104 |
| 130 | +/src/msbuild | 145 |
| 131 | +/src/newtonsoft-json | 25 |
| 132 | +/src/nuget-client | 242 |
| 133 | +/src/roslyn | 152 |
| 134 | +/src/roslyn-tools | 152 |
| 135 | +/src/sdk | 109 |
| 136 | +/src/standard | 18 |
| 137 | +/src/templating | 155 |
| 138 | +/src/vstest | 185 |
| 139 | +/src/websdk | 140 |
| 140 | +/src/xliff-tasks | 131 |
| 141 | +/Tools | 18 |
| 142 | +/tools-local/tasks | 97 |
0 commit comments