Revisiting the release process

[bthomee]

The release process was last discussed a few years ago, and we would now like to refine it even further to improve how we release code.

Summary

• We propose to revise our release process and going forward leverage major, minor, and patch releases:
  • Major releases (X.0.0) will only include changes requiring an amendment.
    • They are released approximately quarterly.
    • PRs are merged into the develop_major branch.
    • Releases are prepared in the release_major branch.
  • Minor releases (X.Y.0) will only include changes not requiring an amendment.
    • They are released approximately every 3 weeks.
    • PRs are merged into the develop_minor branch.
    • Releases are prepared in the release_minor branch.
  • Patch releases (X.Y.Z) will only contain hotfixes and may require an amendment if needed.
    • They are released only when needed to fix a serious bug.
    • PRs are merged directly into the master branch.
• Betas and release candidates are cut from the release branches, whereas releases and hotfixes are cut from the master branch. After any kind of release has been made, the code is merged down to lower branches.
• To reach the desired release cadence and improve code quality, we will leverage GitHub CI to automate the majority of the currently manual release process.

1 Introduction

At the moment a new rippled release is made approximately every quarter, typically varying between 2 and 4 months. A standard release contains one or two new features, as well as a slew of performance and other improvements. There is also usually a mad dash to get a few important or nice-to-have PRs merged in the last few days before a release is cut, since otherwise their functionality won't be available until the next quarterly release.

Around a month is reserved for intensive testing after the release has been cut, to ensure there are no regressions, before the packages are distributed and the release is announced. Sometimes last-minute urgent fixes are merged into the release after it has been cut but before it is published. In practice, the testing time is about 1-2 weeks.

This document proposes to modify the release process to improve the release frequency and better safeguard code quality. In particular, this is achieved by separating non-amendment changes (fast release cadence) from amendment changes (slow release cadence), whereby the former can be trivially rolled back.

2 Proposed solution

We propose to keep changes requiring an amendment separate from all other changes. The former will be included in major (X.0.0) releases, while the latter will be included in minor (X.Y.0) releases. Hotfixes are an exception to the rule and serve to address a newly discovered critical issue, which may or may not require an amendment; these are included in patch (X.Y.Z) releases.

An example release schedule is shown in Figure 1, where every three weeks a minor release is cut, which then via manual and automated testing is published about two weeks later. However, every few months a major release is made, during which the minor releases are temporarily suspended. As major releases carry greater risk, three weeks are reserved for testing.

Figure 1. Example release schedule with circles on the dates indicating minor releases (solid green) and major releases (dashed red), with the cut dates in a lighter color and the release dates in a darker color. For instance, in this example schedule a major release would be cut on 17 June, which would then be made available on 7 July, while a minor release would be cut on 8 July, which would then be made available on 21 July. Although the major release date and the minor cut date are just one day apart, as development continues while a major release is being prepared, there should be sufficient changes to include in a new minor release. Note that the calendar above is for illustration purposes only, and does not account for public holidays except for the vacations around the end of the year.

The motivation for not making a new minor release while a major release is already being prepared but not yet published is because the testing process of the major release would then be evaluating non-final code. Namely, the new code in the minor release would not initially be included in the branch used for the major release, but it would still need to be included before the major release is actually made, possibly introducing bugs due to the untested interaction of the combined changes.

The release schedule can remain flexible, and can be easily adjusted as needed. For instance, if a feature is running behind, the major release can be delayed by a few weeks, possibly allowing for a new minor release to be made in the meantime, while if no meaningful new functionality has been merged then a minor release can be skipped.

2.1 Advantages

The advantages of this proposal are as follows.

2.1.1 Frequent improvements

By making more frequent releases, non-amendment improvements become available more quickly to the public and especially node operators.

2.1.2 Fewer changes

By making more frequent releases, fewer changes are included in a release. This not only allows for the release testing cycles to be shortened, but also enables them to better target the modified functionality. In contrast, at present tens of PRs can be included in a major release, which can modify all aspects of rippled and thus requires extensive and exhaustive testing.

2.1.3 Catch the next minor release

Non-amendment changes that are not ready when a minor release is cut can simply go into the next one a few weeks later. This avoids the rush to get changes in, which otherwise increases the risk of bugs and the creation of tech debt.

2.1.4 Skip minor releases

UNL validators, who tend to not want to update as often given their outsized impact on the XRPL, can focus on the major releases and skip as many minor releases as they wish – effectively maintaining the status quo. We may nonetheless still encourage them to install a minor release when that has important performance or security implications. In general, any node operator can choose to update to or skip a minor release as they see fit; however, everyone must update to a major release to avoid becoming amendment-blocked.

2.1.5 Rolling back minor releases

At present all changes go out together in a single release, which means that if one of the non-amendment changes introduced a bug we cannot roll back due to the presence of amendment-related code, in which case we can only fix forward. Fixing forward, however, may incur a two week voting delay if the fix requires another amendment during which the bug can be exploited. By separating these types of changes we can worry less, since minor releases can be trivially rolled back.

2.2 Disadvantages

The disadvantages of this proposal are as follows.

2.2.1 Automating the release process

Our current release process is mostly manual. To reduce the burden on the maintainers when making a new release every few weeks, the process must be automated as much as possible. This includes (i) turning the code into artifacts, (ii) testing the artifacts, and (iii) releasing the artifacts, and (iv) automating generation of the release notes. Although automating the release process will be a good thing, getting to that point will demand a considerable time investment.

2.2.2 Juggling branches and commits

Making more frequent releases requires juggling more branches. This will involve more extensive use of git, including cherry-picking commits, reverse-merging, and resolving conflicts, which necessitates creating new processes for the maintainers to follow. However, we aim to provide helper scripts to cover the vast majority of situations, and leave the rest to automations via GitHub CI.

2.2.3 Upgrade fatigue

Communication about the releases will need to be improved to avoid update fatigue, for those not taking advantage of the auto-update scripts. In that situation, node operators will need to be notified through other channels so they can perform the update. In particular, it is easy to notice an announcement every quarter, while if they happen every two to three weeks they may be easier to miss, or node operators may simply skip them as they may think the benefits do not outweigh the effort of updating.

2.2.4 Release notes and documentation

The new release process will demand more time for preparing release notes and documentation. As new features will only be released every quarter, which require updated documentation, we do not expect a significant increase in burden. However, occasional performance / architecture improvements that will be published as a minor release may warrant updates to the documentation. Release notes are already mostly auto-generated via GitHub.

2.2.5 Close collaborations with other XRPLF developers

With more frequent releases there may be a need for closer collaboration with the developers responsible for Clio,xrpl.js, etc., especially in terms of aligning changes made to their codebases and ensuring their release schedule matches ours for important changes.

2.2.6 Challenging troubleshooting

Once we start releasing more frequently, it is likely we will have a diversity of versions running on the network, both UNL validators and non-UNL validators. As a result, when an issue surfaces it may become more difficult to pinpoint the root cause.

2.3 Discussion

By investing some time in (i) defining the branching strategy, (ii) automating our release process, (iii) educating the repository maintainers, and (iv) creating and following a tentative release schedule, we can reap the rewards of releasing faster, improving code quality and testability, reducing tech debt, and giving us more room to maneuver in case a bug is detected.

Properly executing the new release process will require giving the maintainers the strength to say "no", especially when there are incentives or pressure to deviate from the new process "to include just this one small / important PR" in a release. We expect that initially, as the new release process is being tried out, there will still be changes included in a major release that ought to have been included in a minor release. However, we also expect that over time, as the maintainers get used to the new process, the releases will get cleaner and the time between cut and publish can be shortened to release even more frequently.

3 Alternatives considered

3.1 Backward-(in)compatible changes

We can change our definitions of what is included in major and minor releases by focusing on backward-compatibility instead. In this option, backward-compatible changes are excluded from major releases, while all backward-incompatible changes are excluded from minor releases. This ensures that minor releases can always be rolled back. A major release will then still typically introduce one or more amendments while a minor release will never do so, as before. However, it is possible for a major release to not introduce any amendment, for instance when it contains a major "breaking" refactor of fundamental logic but no new features.

Discussion

While this option sounds appealing, by including non-amendment backward-incompatible changes in a major release, we may increase the amount of code that is part of a major release and would need to be tested. In contrast, in the proposed solution a backward-incompatible change can be released as a separate minor release, to allow for targeted testing of just that functionality.

Furthermore, it is not clear if changes to rippled can even be backward-incompatible. Since nodes upgrade to a new version independently, new releases must therefore be compatible with old releases. In particular, one can just wipe all data used by rippled, roll back to the previous release, and receive the data again from all peers.

3.2 More frequent minor releases

This option is a hybrid between the status quo and the proposed solution, whereby minor releases are frequently made containing non-amendment changes, while major releases would also be allowed to contain non-amendment changes. This option would use the same release schedule as the proposed solution.

Discussion

An advantage of this option is that non-amendment changes won't have to be withheld from major releases, so development can progress more naturally.

A disadvantage is that a bug introduced by a non-amendment change included in a major release cannot be rolled back and only fixed forward, as is currently the case. Moreover, like the previous option proposed above, by including non-amendment changes in a major release, we increase the amount of code that is part of the release and needs to be tested.

3.3 Cherrypicking

Rather than juggling multiple branches, we can keep things simple with a develop, release, and master branch like now. When it is time to cut a beta or release candidate for a major or minor release, we cherrypick the corresponding commits from the develop branch into the release branch, and when it is time to make a release we promote the latest release candidate into the master branch.

Discussion

Although the seeming simplicity of cherrypicking is appealing, in reality it may be difficult to do due to (i) having to tease apart the code applicable to minor releases from the code applicable to major releases, (ii) conflicts can occur in when the cherrypicked changes are subsequently changed again, and (iii) no conflicts occur when they should, resulting in code differing from what one would intuitively expect; see examples here and here.

Furthermore, we may not be able to complete a minor release when it is time for a major release. The release branch will then already have one or more betas or release candidates containing non-amendment changes, which will pollute the major release containing only amendment changes we now want to make instead. In sum, cherrypicking is more trouble than it's worth.

3.4 Feature branches

In this option, no amendment code is merged into the develop branch until it is time to make a major release, allowing us to keep the existing branch structure with release and master. Non-amendment code will be continuously merged into the develop branch, enabling us to make minor releases every few weeks. It is the responsibility of the developers to frequently merge the develop branch into their feature branches to keep them up to date, which may require resolving conflicts from time to time.

Discussion

Code requiring an amendment is typically kept in long-lived feature branches, which already requires their developers to keep them up to date relatively frequently - the main difference proposed by this option would be that each feature branch is not merged until closer to the release date.

A typical release only includes a handful of features, so the amount of work to keep the code in the branches will generally be relatively minor. Even though minor, keeping the code in branches instead of merging them does require non-trivial effort - busywork to be precise - that could be better spent on other more useful things. Moreover, when the release date is coming up, merging multiple feature branches that have been independently developed could result in new conflicts that need to be resolved, as well as unexpected interactions between these features that may be difficult to detect and debug so close to the release date.

3.5 Do nothing

The current release process works well enough, and is not worth the time and effort to improve. Instead, we should concentrate on bringing more features to the ledger, reducing tech debt, reviewing the 60+ open PRs, and addressing the 350+ open issues.

Discussion

This document has highlighted various shortcomings of the current process, such as it consuming a lot of time from maintainers, releases containing many changes that require extensive testing, delayed availability of improvements, and so on. We believe that a short-term investment in an improved release process will in the long-term free up the developers to a larger extent, which then can be used to introduce new features, reduce tech debt, etc.

4 Implementation

Betas and release candidates are currently made by copying the latest code from the develop branch into the release branch, updating the version number, and releasing it from there. Releases are made by copying that code into the master branch, updating the version number once again, and then releasing it from there. Afterwards changes are merged down to ensure all lower branches have the released code. Hotfixes are made in the master branch, and merged down afterwards.

To separately release amendment and non-amendment code, we need to slightly change our branching and release strategies. We further wish to automate the process as much as possible to reduce toil.

4.1 Terminology

Contributor

A person who creates a pull request (PR) from their private fork into the rippled repository.

Developer

A person who can create a PR from a branch in the rippled repository, which requires Write access. A developer has a superset of permissions relative to a contributor.

Maintainer

A person who can merge a PR into a protected branch, which requires Maintain access. A maintainer has a superset of permissions relative to a developer.

4.2 Branching strategy

We will distinguish between development intended for minor releases from development intended for major releases by having two parallel sets of protected branches that culminate in the master branch, see Figure 2.

Figure 2. Illustration of the branching setup. Protected branches are shown in yellow (center), while regular branches are shown in purple (top), green (bottom-left), or salmon (bottom-right) depending on whether they target a patch, minor, or major release, respectively. Changes are pushed up from the bottom minor and major branches into the higher branches, where betas and release candidates are made from either release branch, while releases are made from the master branch. After a release has been made from the master branch, the changes are merged back into the lower branches that do not yet have them. Hotfixes are merged directly into the master branch before making a patch release.

Our branching strategy uses the following protected branches.

The master branch

The master branch will contain the latest released code, excluding betas and release candidates. When a major, minor, or patch release has been made the code is merged down into all release and develop branches that do not yet have those changes.

The release_minor branch

The release_minor branch will contain the code for the next pending minor release. No code will be pushed to the master branch until we are ready to make a minor release.

The release_major branch

The release_major branch will contain the code for the next pending major release. No code will be pushed to the master branch until we are ready to make a major release.

The develop_minor branch

The develop_minor branch is where development is done that leads to minor releases. This branch will be the default branch, so any created PR will use it as the base branch by default. No code will be pushed to the release_minor branch until we are ready to make a beta or release candidate.

The develop_major branch

The develop_major branch is where development is done that leads to major releases. As amendment code is written less frequently than non-amendment code, the authors of any amendment PRs will need to manually select the develop_major branch as the base branch, which is a trivial step. No code will be pushed to the release_major branch until we are ready to make a beta or release candidate.

Hotfixes

Hotfixes are a special case and are released only when needed to fix a serious bug, and result in the creation of a patch release. A hotfix will be created on a separate branch that is based off the latest code in the master branch, since the release branches may have moved on. Once merged into the master branch, it is merged down into all release and develop branches. A hotfix should not need a beta or release candidate, as it should be a very targeted change that can be tested separately.

In Figure 3 we show a different perspective of how the code flows between the branches, and how they are tagged.

Figure 3. An illustration of the proposed merging strategy. The green checkmarks represent PR merges into the develop branches, while the orange targets represent PR merges into the release or master branches that result in some type of release; the latter events are accompanied by a purple flash to highlight where the code came from. When a bug is found, a patch release is made. After each release the code is merged back from the master branch into all release and develop branches, indicated by the grayish double-arrows.

4.3 Development strategy

The development process described below is shown in Figure 4.

4.3.1 Local changes

To make a change to the codebase, a developer first starts locally with a fresh branch based on the target branch.

Amendment changes

New features or other modifications that require an amendment will be merged into the develop_major branch.
It is the responsibility of the developer to make sure their local branch is periodically brought up to date with the develop_major branch.

Non-amendment changes

Any modification that does not need an amendment will be merged into the develop_minor branch.
It is the responsibility of the developer to make sure their local branch is periodically brought up to date with the develop_minor branch.

Betas and release candidates

Before proposing a beta or release candidate, the developer must bring their local branch, which is based on the release_major branch for major releases or the release_minor branch for minor releases, up to date with the commits to include. This is usually achieved by pulling in the latest changes from the corresponding develop_major or develop_minor branch, resolving conflicts as needed.

To allow development to continue uninterrupted and to remain flexible when a code improvement is delivered in parts (in particular, we prefer many small PRs over few large PRs) there may be commits that we wish to skip, in which case we need to cherrypick the ones to include. Given the aforementioned challenges with cherrypicking, care must be taken when doing so.

Releases

Before proposing a release, the developer must bring their local branch, which is based on the master branch, up to date with the commits to include. This is achieved by pulling in the latest changes from the release_major branch for major releases or the release_minor branch for minor releases, resolving conflicts as needed.

In general, the last release candidate from the appropriate release branch contains the code that will be released, in which case no local changes actually need to be made. However, hotfixes are a special case and will require local development.

4.3.2 Remote changes

Commits

When pushing a new commit to a PR, our GitHub CI workflow will execute the regular GitHub CI pipeline to build and unit test the code on Linux, MacOS, and Windows; the latter two platforms are best-effort only, i.e. we will ensure the code compiles and runs, but they are not officially supported. To merge a PR one or two approvals are needed, depending on how trivial the change is.

Merges

When a PR is merged into one of the protected branches, our GitHub CI pipeline will create the necessary Linux packages and Docker images tagged with the (shortened) commit hash, and will publish them to our internal registry. The pipeline will then set up a small yet realistic network on a test bench and execute a suite of integration tests to confirm both performance and correctness; the suite will be more comprehensive on Linux than on MacOS and Windows. PRs are not permitted to be merged into the target branch if they are out of date.

Note: Currently the BuildInfo.cpp file contains the release version, which is a painful part of the current release process as a new commit is needed to update it, which then kicks off an entire build and test cycle. To avoid this, we will follow the lead of Clio, whereby the version is determined using Git and then injected into the BuildInfo.cpp file by CMake before the binary is rebuilt. Since only the version changes, we can skip running tests.

Note: Currently the release notes are added to RELEASENOTES.md when making a release, in addition to being visible in GitHub and also included in the accompanying blog post published by our communications team afterwards. To avoid having to make an arguably superfluous change in the master branch that then needs to be merged down into all release and develop branches, we will remove this file going forward to save us time and effort.

QA

Any additional testing that targets the merged changes can be performed manually at this stage. This will typically only be necessary in the release and master branches. We further would like to ensure that no one rebuilds the binary themselves for QA testing to avoid accidental mistakes, such as testing a different version of the code. This is why the testbed will only run using the internally hosted artifacts. As the testbed exports logs and metrics to Grafana (and in the future traces as well), they can be analyzed for regressions.

Devnets

In addition to deploying to the testbed, we may consider automatically deploying betas and release candidates to devnet to be exposed to more realistic traffic. However, as major releases cannot be rolled back once amendments have been activated – unless devnet is fully reset – we require the voting to be done manually.

We must use two separate devnets, because the release_minor and release_major branches will be developed independently. Specifically, they will diverge over time and only come back together again after each release, while new amendments proposed in the release_major branch would not be available in the release_minor branch.

To illustrate, if we were to use a single devnet then we could not first deploy a minor release candidate, e.g. 4.5.0-rc1, then a major release candidate, e.g. 5.0.0-rc1, and finally deploy another minor release candidate, e.g. 4.5.0-rc2, without resetting devnet in between due to the new amendments introduced by the major release candidate that the minor release candidate would not have and get blocked by.

A hotfix, depending on whether they require an amendment, should be deployed on the appropriate devnet.

Figure 4. GitHub CI pipelines are triggered when adding a commit or merging a PR.

4.4 Release strategy

As the release process is quite involved and mistakes are easy to make when manually executing commands, we will automate the process to the maximum extent possible. In particular, we will create two scripts that perform the heavy lifting.

Betas and release candidates

The first script triggers a pipeline as illustrated in Figure 5.

Figure 5. GitHub CI pipelines are triggered when creating a beta or a release candidate from a release branch.

Pipeline

• Requires as inputs:
  • The commit hash on the release_major or release_minor branch.
  • The tag to apply.
• Performs three initial checks:
  • If the tag matches the beta or release candidate semantic versioning.
  • If the tag is next-in-line, which essentially means:
    • The tag does not yet exist.
    • For major releases, i.e. the tag matches X.0.0-(b|rc)N.
    • For betas, i.e. the tag matches X.0.0-bN.
      • X.0.0-b1 if (X-1).Y.Z is the latest release.
      • X.0.0-bN if X.0.0-b(N-1) is the latest beta release.
    • For release candidates, i.e. the tag matches X.0.0-rcN.
      • X.0.0-rc1 if (X-1).Y.Z is the latest release.
      • X.0.0-rcN if X.0.0-rc(N-1) is the latest release candidate.
  • For minor releases, i.e. the tag matches X.Y.0-(b|rc)N.
    • For betas, i.e. the tag matches X.Y.0-bN.
      • X.Y.0-b1 if X.(Y-1).Z is the latest release.
      • X.Y.0-bN if X.Y.0-b(N-1) is the latest beta release.
    • For release candidates, i.e. the tag matches X.Y.0-rcN.
      • X.Y.0-rc1 if X.(Y-1).Z is the latest release.
      • X.Y.0-rcN if X.Y.0-rc(N-1) is the latest release candidate.
  • If the commit actually exists on the branch.
    • If the tag matches X.0.0-(b|rc)N then the release_major branch is checked, otherwise the release_minor branch is checked.
• Applies the tag to the new commit.
  • This requires approval using the deployments feature of GitHub.
  • Developers can approve betas and release candidates.
• Rebuilds the artifacts using the new tag and publishes them to our external registry.
  • This allows the community to test rippled.
  • If the tag points to a release candidate, auto-generates release notes and publishes a pre-release. Betas are not published.

In practice we may split the script into two scripts to distinguish between minor and major releases to avoid some of the tag matching logic described above.

Releases

The second script triggers three to five pipelines in sequence and is illustrated in Figure 6.

Figure 6. GitHub CI pipelines are triggered when creating a release from the master branch. Depending on the type of release some of the PR pipelines can be skipped as they would be no-ops.

Pipeline 1

• Requires as inputs:
  • The commit hash on the master branch.
  • The tag to apply.
  • (optional) The release notes to publish.
    • If not provided, they will be auto-generated.
    • If provided and they contain the Jinja {{Generated}} stanza, release notes will be auto-generated and inserted at that location.
• Performs three initial checks:
  • If the tag matches the release semantic versioning.
  • If the tag is next-in-line, which essentially means:
    • The tag does not yet exist.
    • (X+1).0.0, X.(Y+1).0, or X.Y.(Z+1) with X.Y.Z being an existing release.
      • By not requiring the existing release to be the latest version, we are able to support hotfixes that modify an earlier release.
    • If the commit actually exists on the branch.
• Applies the tag to the new commit.
  • This requires approval using the deployments feature of GitHub.
  • Only maintainers can approve releases.
• Rebuilds the artifacts using the new tag and publishes them to our external registry.
  • Creates and/or generates the release notes, and publishes the release.

Pipelines 2, 3, 4, and 5

• Create PRs to push the changes from the master branch down into all release and develop branches.
  • As the branches may have advanced relative to the master branch, some conflicts may need to be manually resolved.
    • Not all pipelines need to run, depending on the type of release, as the changes only need to be merged into the branches that do not have them yet.
      • For patch releases: all release and develop branches.
      • For minor releases: only the release_major and develop_major branches.
      • For major releases: only the release_minor and develop_minor branches.
    • Each PR can be set to auto-merge after approval.

In practice we may split the script into three scripts to distinguish between patch, minor and major releases to avoid some of the tag matching logic and simplify the PR pipelines described above.

5 Conclusion

By separating development into amendment and non-amendment changes we will be able to make more frequent releases, which are smaller and therefore easier to comprehensively test, which ultimately should increase code quality, reliability, robustness, and security.

By further automating the process as much as possible we would greatly reduce the reliance on manual effort to create betas, release candidates, and releases, leaving developers with more time to spend on value-add contributions.

Some manual efforts would be needed after a release has been made to update lower branches, but we believe that with smaller releases the efforts to resolve conflicts and to merge down the changes will be manageable.

[bthomee]

To ease into the new release process, we propose a simpler approach to initially follow until the end of this year, after which we can reassess whether to proceed with the new release process or whether the simpler approach is good enough.

• The approach involves maintaining the current develop and release branches, while continuing to merge features using the Supported::no stanza to keep them disabled.
  • This allows us to make frequent minor releases (X.Y.0) containing non-feature changes, while any new feature code included in the release will be inactive.
  • When it is time to make a quarterly major release (X.0.0), the features will be updated to use the Supported::yes stanza.
  • There will be a mixture of non-feature and feature code in major releases, but since non-feature code is released frequently in minor releases, there will not be too many such changes.
  • Big non-feature changes that carry some risk will be held back until the next minor release, even if they are ready for inclusion in a major release.

[mvadari]

When will this start? Do PRs slated for 2.5.0 need to be adjusted?

[bthomee]

We're looking to start this after 2.5.0, so nothing will change until then. Once 2.5.0 is out, the only difference will be setting Supported::no for features if they are merged well before a major release is scheduled.