Skip to content

Commit 253165d

Browse files
dagooddagood
authored
Draft a Servicing Granularity overview doc (#1389)
* Draft ServicingGranularity doc * Move doc alongside arcade-sb doc, and add hook
1 parent 87b7061 commit 253165d

File tree

3 files changed

+229
-0
lines changed

3 files changed

+229
-0
lines changed

‎Documentation/planning/arcade-powered-source-build/README.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,9 @@ build process is driven by two major goals:
4242
impact of changes, reducing the frequency the source-build servicing team
4343
has to root-cause and patch over problems.
4444

45+
* Gives us a place to add additional checks in the future, such as
46+
[nongranular servicing readiness](../nongranular-servicing-readiness/).
47+
4548
This doc is about where we can start, what incremental progress would look like,
4649
and the vision.
4750

‎Documentation/planning/nongranular-servicing-readiness/README.md

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# Non-granular servicing readiness
2+
3+
This doc describes the risks caused by servicing granularity differences between
4+
Microsoft and source-build, and proposes reducing the risk by adding tests to
5+
the Microsoft build ensuring source-build's lack of granularity doesn't cause
6+
problems. Alternative approaches are discussed in
7+
[ServicingGranularity-RejectedApproaches.md](ServicingGranularity-RejectedApproaches.md).
8+
9+
This is related to [source-build#923 "Figure out how source-build will work with
10+
CoreFX per-package servicing
11+
policy"](https://github.com/dotnet/source-build/issues/923).
12+
13+
## "Build only if changed" policy
14+
15+
This is a maintenance strategy where if a *product* hasn't changed, it isn't
16+
rebuilt and won't be rereleased under a new patch version.
17+
18+
> A *product* for the sake of this doc is any released asset where an
19+
independent decision is made whether that specific asset will be patched and
20+
shipped.
21+
22+
For projects that release NuGet packages, it's ordinary to treat each NuGet
23+
package as a product. For example, when a new .NET Core SDK patch is being built
24+
to take a networking stack fix, the Microsoft servicing workflow doesn't involve
25+
building and releasing new MSBuild `.nupkg` files to the NuGet Gallery. When the
26+
patched SDK is built, old MSBuild binaries are retrieved from the NuGet Gallery.
27+
28+
Linux distributions tend to follow the same strategy, but the product is the set
29+
of distro packages built from a single source package. To create a distro
30+
package release, the entire source package is built. If one of the patched
31+
distro packages takes a dependency built by a different source package, that
32+
source package doesn't need to be rebuilt: the old build of the dependency is
33+
used.
34+
35+
The difference is that the Microsoft .NET Core SDK build is very granular: each
36+
NuGet package may be a product, even if it's built from the same source tree as
37+
other NuGet packages. This causes a conflict when Microsoft chooses to build and
38+
release a subset of what it considers the products contained in a specific
39+
source package, but Linux distro expectations are to release the entire product
40+
(all packages) from that source package. The partial build behavior is the
41+
default when building from a `vX.Y.Z` Git tag in a dotnet repository.
42+
43+
## How source-build works now, despite different granularity
44+
45+
The current approach is to disable the partial build behavior through a build
46+
flag while building the tag.
47+
48+
🚩 This is a risky workaround with the current state of the repos. There's no
49+
mechanism in place that ensures the new packages will behave the same as the
50+
ones produced by earlier Microsoft releases. See [source-build#923 "Figure out
51+
how source-build will work with CoreFX per-package servicing
52+
policy"](https://github.com/dotnet/source-build/issues/923).
53+
54+
## Can we unify servicing behavior?
55+
56+
In general, we want to minimize differences between the Microsoft build and
57+
source-build. Making the builds behave the same way would be the natural way to
58+
resolve the issue.
59+
60+
Applying the Microsoft build approach to source-build, we could split the
61+
source-build .NET Core SDK into many source packages (and in turn, many distro
62+
packages). This is considered unmaintainable. (See ['Switching to microsoft
63+
build
64+
granularity'](ServicingGranularity-RejectedApproaches.md#switching-to-microsoft-build-granularity).)
65+
66+
The other direction, making the Microsoft build use source-build's less granular
67+
policy, is likely a non-starter. It would force unnecessary redownload of
68+
products with no meaningful changes, a regression affecting products such as
69+
Visual Studio.
70+
71+
Neither direction appears to be viable.
72+
73+
## Reduce the risk and continue disabling partial build behavior
74+
75+
We can continue with how things are now, but add tests to validate the different
76+
approaches don't cause incompatibilities.
77+
78+
We should run the usual tests against the source-build outputs, to know the SDK
79+
and runtimes function as expected. This is a base level of validation.
80+
81+
However, that would miss compatibility issues between source-build and the
82+
Microsoft build. For example, if source-build produces a 3.0.2 version of some
83+
assembly, but Microsoft hasn't released a patch to that assembly since 3.0.0, it
84+
may be possible for a dev to publish a project that depends on the 3.0.2 binary
85+
and fails to run on the Microsoft .NET Core Runtime because it only has 3.0.0.
86+
87+
Additional testing may be able to cover compatibility problems. If the .NET Core
88+
repos know the version Microsoft last shipped of each product (already the case
89+
in some repos), tests can download the last Microsoft-shipped packages and
90+
compare the contents to the current. Tests can also analyze source code diffs to
91+
spot suspicious unreleased changes.
92+
93+
Reversing the perspective may help clarify. Microsoft has an atypical servicing
94+
policy where some source packages are not fully built. The proposal is to add
95+
tests that help ensure compatibility when Microsoft chooses not to build part of
96+
a source package.

‎Documentation/planning/nongranular-servicing-readiness/ServicingGranularity-RejectedApproaches.md

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# Servicing Granularity - Rejected Approaches
2+
See [README.md](README.md) for context on the servicing granularity problem.
3+
This document discusses rejected approaches.
4+
5+
# Switching to microsoft build granularity
6+
This section discusses the consequences of switching to the very granular
7+
Microsoft servicing policy.
8+
9+
## Background: granularity of the Microsoft servicing build
10+
How granular is the Microsoft build really? Each dotnet repo is generally a
11+
product, but some repositories go further, containing multiple products in a
12+
single source tree. This is an example based on 3.0 (some products may be
13+
missing from the list):
14+
15+
* CoreFX
16+
* The NETCoreApp shared framework implementation (product)
17+
* `runtime.linux-x64.Microsoft.Private.CoreFx.NETCoreApp`
18+
* Always built when NETCoreApp gets any patch.
19+
* The out of band (OOB) packages (*each* is a product)
20+
* `System.Data.SqlClient`
21+
* `System.Diagnostics.EventLog`
22+
* `System.Drawing.Common`
23+
* `System.IO.Pipelines`
24+
* `System.IO.Ports`
25+
* ...more. These `System.*` packages are produced in CoreFX, but aren't
26+
included in the NETCoreApp framework. Some are used by downstream
27+
frameworks like ASP.NET Core, and by the tools and SDK. They are also used
28+
by apps directly via `PackageReference`.
29+
* These are somewhat likely to get patches over time.
30+
31+
* Core-Setup
32+
* The NETCoreApp shared framework and Runtime/AppHost pack (product)
33+
* `Microsoft.NETCore.App.Runtime.linux-x64`
34+
* Always built when NETCoreApp gets any patch.
35+
* The NETCoreApp Targeting pack (product)
36+
* NETStandard Targeting pack (product)
37+
* Targeting packs are unlikely to get patches.
38+
39+
* AspNetCore
40+
* ASP.NET Core shared framework and Runtime pack (product)
41+
* `Microsoft.AspNetCore.App.Runtime.linux-x64`
42+
* ASP.NET Core Targeting pack (product)
43+
44+
This means if you assemble a dependency graph of a theoretical .NET Core SDK,
45+
say 3.0.103, it will contain multiple versions of the same repos:
46+
47+
* .NET Core SDK `v3.0.103`
48+
* CoreFX
49+
* `v3.0.3` - System.Security.Permissions, shared framework implementation
50+
* `v3.0.2` - System.ServiceProcess.ServiceController
51+
* `v3.0.1` - System.Security.Cryptography.Xml
52+
* `v3.0.0` - System.IO.Pipelines
53+
* [...]
54+
55+
This shows how the number of nodes will naturally grow over time. The max number
56+
of versions of a specific repo is bounded by the number of products it builds.
57+
The max is only three for Core-Setup, but larger (dozens?) for CoreFX because of
58+
the number of independent OOB packages.
59+
60+
## Consequences of depending on multiple versions of the same repo
61+
62+
### Microsoft builds
63+
This is completely fine for Microsoft servicing builds, because they use
64+
prebuilts. When building a repo, all upstream products are downloaded from an
65+
online source, so the *version* of each product isn't important.
66+
67+
### Building from source
68+
Multiple versions of the same repo has significant consequences for
69+
source-build, on the other hand. Downloading prebuilt products is forbidden, so
70+
every upstream product must be built from source. The more commits built, the
71+
longer the build takes.
72+
73+
In some cases, we can redistribute the previous source-built binaries, but we
74+
have to account for bootstrapping on new distros (no previous source-build
75+
exists) and distros that don't keep old binaries around:
76+
77+
> Some (not all) Linux distribution do keep around older distro packages in
78+
their distro repository (eg, we build source-build 3.0.10, and 3.0.9 is still
79+
around for installation). If that was a general policy (it's not, for example,
80+
in Fedora) than source-build could do what Microsoft build does and use older
81+
builds. Aside from being not being supported everywhere, it also means
82+
source-build needs previous patch versions to have been built in that distro
83+
too. A new distro couldn't skip packaging 3.0.0 if 3.0.10 still needs something
84+
from there. [[omajid's review
85+
comment](https://github.com/dotnet/source-build/pull/1389#discussion_r350783942)]
86+
87+
Even worse, the number of dependency nodes will tend to increase over time as a
88+
branch gets serviced, as demonstrated above with the list of tags comprising a
89+
theoretical `v3.0.103` SDK. This means the difficulty and complexity of building
90+
.NET Core would then depend on how many times the branch has had a servicing
91+
release.
92+
93+
A few concrete reasons this causes significant overhead and/or upfront work for
94+
source-build infra maintainers:
95+
96+
1. Say a new distro appears that can't build anything earlier than CoreFX
97+
`v3.0.3`, but we need to bootstrap .NET Core. To build the required `v3.0.0`,
98+
`v3.0.1`, and `v3.0.2` tags, we must add patches to each one to get them to
99+
build. The patching work *probably* only needs to be done once and applied on
100+
all affected commits, but as the release branch continues to be serviced, the
101+
amount of work to get the product built on a new distro compounds.
102+
2. Each repo must be checked out at every required commit/tag, which syncs a
103+
*lot* of irrelevant source code.
104+
3. The repo must either be built in its entirety, or we need significant infra
105+
work to allow source-build to pick the exact product it needs to build.
106+
107+
## Q&A
108+
109+
### Can we split up source-build products to make this less difficult?
110+
The thought here is that we might be able to split source-build into more
111+
tarballs with more intermediate distro packages. This could reduce the amount of
112+
code being compiled at once, making the build more incremental.
113+
114+
There is direct feedback from experienced distro maintainers that we need to
115+
keep it simple: continue building a ".NET Core SDK + Runtime" product. Asking
116+
distro maintainers to keep track of many products is a significant burden and
117+
expected to cause mistakes.
118+
119+
This also doesn't solve (1) from above. We would still need to add patches to
120+
old, unmaintained tags to bring up .NET Core on a new distro.
121+
122+
### Does source-build-reference-packages help?
123+
Context: any NuGet package that only contains references can be built by
124+
[dotnet/source-build-reference-packages](https://github.com/dotnet/source-build-reference-packages)
125+
rather than checking out and building the original repo. In some cases, such as
126+
targeting packs, we already do this.
127+
128+
This doesn't work for CoreFX OOB packages, which contain implementation. The OOB
129+
packages are a particular pain point with this approach, so
130+
source-build-reference-packages doesn't help significantly.

0 commit comments

Comments
 (0)