Skip to content

Commit bb1a20c

Browse files
adwk67maltesanderrazvan
authored
Create release branch/tag scripts (#35)
* wip * wip create release branch * wip push changes * wip release tags * wip: replace dry-run flag with push flag * removed tags (just creating branch) * wip updates * changed regex to be semver compatible * added regex check * fix indentation * include code changes and call templating script * wip: update spec.version in multiple files * wip * fixed yq processing of multiple files * add checks to branch script callout * wip: scripts for branching and tagging * added changelog/.j2 updates * added some explanation * typo * improve argument handling * removed superfluous file * added new documentation * read repo and operators from config.yaml * use test images repo * corrected var resolution * added tags push * added note to docs about filtering build actions * tested and updated create-release-branch.sh * added update branch command * wip scripts * fix -e * minor fixes * add/re-assign docs tag * remove release script and revert docs tag * refine/fix example yaml/kuttl test files * Some updates to bulk pr handling. * First successful run of the release scripts. * update includes statement in doc files * Update README.adoc * Update create-release-branch.sh Remove duplicate readme. * Update create-release-tag.sh Remove duplicate readme * Update config.yaml Complete list of operator repos. * update utils image * bugfixes * update test-definition.yaml instead of templates * Introduce the "--what" option for release scripts. * sorted operator names * Fix command to push tag * fixed antora version to be non-patch-level * Do not break getting started yamls when patching stackableVersion Co-authored-by: maltesander <malte.sander.it@gmail.com> Co-authored-by: Razvan-Daniel Mihai <84674+razvan@users.noreply.github.com>
1 parent fd3c029 commit bb1a20c

14 files changed

+3159
-159
lines changed

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
/.idea

‎bulk-pr/README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,3 +54,7 @@ The displayed status per PR can have the following values:
5454
|--------|------------------------------------------------|
5555
| 0 | All checks were finished and successful |
5656
| 1 | One or more checks failed or are still running |
57+
58+
### pr_list
59+
List all open PRs in all `repos`.
60+

‎bulk-pr/pr_list

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
#!/usr/bin/env bash
2+
source repos
3+
# Disable gh paging for results.
4+
PAGER=
5+
for product in "${products[@]}"; do
6+
echo "#### PRs for $product ###"
7+
gh pr list -R stackabletech/${product}-operator
8+
done

‎bulk-pr/repos

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ products=(
77
hdfs
88
hive
99
kafka
10+
listener
1011
nifi
1112
opa
1213
secret

‎release/README.adoc

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
# Release scripts
2+
3+
A set of scripts that automates some release steps. The release process has three steps:
4+
5+
1. Call `create-release-branch.sh`- This will:
6+
7+
- create a temporary folder
8+
- clone operator and image repositories
9+
- create a new release branch for each repository
10+
11+
2. Test and fix things in the release branches
12+
13+
- changes should be done by making changes in the main branch and then cherry-picking these changes back into the release branch
14+
15+
3. Call `create-release-tag.sh`- This will:
16+
17+
- checks that the appropriate release branch exists
18+
- switch to the previously-created release branch in each repository
19+
- conduct code refactoring
20+
- commit and tag these changes
21+
- push the resulting commits and tags, triggering github actions to build the product images and operators
22+
23+
These steps are summarised in the two diagrams below:
24+
25+
image::images/rb-flow.png[]
26+
27+
image::images/rb-utils.png[]
28+
29+
## Install requirements
30+
31+
### Python 3
32+
33+
A working python 3 installation is required by the `cargo-version.py` script.
34+
35+
Install required packages:
36+
37+
[source]
38+
----
39+
pip install -r release/requirements.txt
40+
----
41+
42+
### yq (yaml parser)
43+
44+
This script requires https://github.com/mikefarah/yq (not to be confused with https://github.com/kislyuk/yq)
45+
46+
## Usage
47+
48+
### Release branches
49+
50+
To create release branches use the `create-release-branch.sh` script, called from the repository root folder. The syntax is given below:
51+
52+
[source]
53+
----
54+
./release/create-release-branch.sh -b <release> [-p] [-c] [-w products|operators|both]
55+
----
56+
57+
- `-b <release>`: the release number (mandatory). This must be a semver-compatible value (i.e. without leading zeros) such as `23.1`, `23.10` etc. and will be used to create a branch with the name `release-<release>` e.g. `release-23.1`
58+
- `-p`: push flag (optional, default is "false"). If provided, the created branches plus any changes made as part fo this process will be pushed to the origin.
59+
- `-c`: cleanup flag (optional, default is "false"). If provided, the repository folders will be torn down on completion.
60+
- `-w`: where to create the branch. It can be "products", "operators", "both".
61+
62+
N.B. the flags cannot be combined (e.g. `-p -c` but not `-pc)
63+
64+
e.g.
65+
66+
[source]
67+
----
68+
./release/create-release-branch.sh -b 23.1 -p -c -w both
69+
----
70+
71+
#### What this script does
72+
73+
* checks that the release argument is valid (e.g. semver-compatible, just major/minor levels)
74+
* strips this argument of any leading or trailing quote marks
75+
* for docker images
76+
** creates or updates a temporary folder with clones of the images repository (given in `config.yaml`)
77+
** creates the new branch according to the release version
78+
** pushes the new branch (if requested with "-p")
79+
** deletes the temporary folder (if requested with "-c")
80+
* for operators:
81+
** iterates over a list of operator repository names (listed in `config.yaml`), and for each one:
82+
** clones or updates operator and product images repositories
83+
** creates the new branch according to the release version
84+
** pushes the new branch (if requested with "-p")
85+
** deletes the temporary folder (if requested with "-c")
86+
87+
88+
### Release tags
89+
90+
To create release tags use the `create-release-tag.sh` script, called from the repository root folder. The syntax is given below:
91+
92+
[source]
93+
----
94+
./release/create-release-tag.sh -t <release-tag> [-p] [-c] [-w products|operators|both]
95+
----
96+
97+
- `-t <release-tag>`: the release tag (mandatory). This must be a semver-compatible value (i.e. major/minor/path, without leading zeros) such as `23.1.0`, `23.10.3` etc. and will be used to create a tag with the name
98+
- `-p`: push flag (optional, default is "false"). If provided, the created commits and tags made as part of this process will be pushed to the origin.
99+
- `-c`: cleanup flag (optional, default is "false"). If provided, the repository folders will be torn down on completion.
100+
- `-w`: where to create the tag and update versions in code. It can be "products", "operators", "both".
101+
102+
N.B. the flags cannot be combined (e.g. `-p -c` but not `-pc)
103+
104+
e.g.
105+
106+
[source]
107+
----
108+
./release/create-release-tag.sh -t 23.1.0 -p -c -w both
109+
----
110+
111+
#### What this script does
112+
113+
* checks that the release argument is valid (e.g. semver-compatible, major/minor/patch levels)
114+
* for docker images:
115+
116+
** tags the branch and pushes it if the push argument is provided
117+
* for operators:
118+
** checks that the release branch exists and the tag doesn't
119+
** adapts the versions in all cargo.toml to `release-tag` argument
120+
** update all "stackableVersion" fields in the getting started guides
121+
** update all "stackableVersion" fields in the kuttl tests
122+
** update the antora.yaml
123+
** update the `release-tag` in helm charts
124+
** updates the cargo workspace
125+
** rebuilds the helm charts
126+
** bumps the changelog
127+
** creates a tagged commit in the branch (i.e. the changes are valid for the branch lifetime)
128+
** pushes the commit and tag (if requested with "-p")
129+
** deletes the temporary folder (if requested with "-c")
130+
131+
#### Build actions
132+
133+
When a tag is pushed, the images for products and operators are built via github actions. The following points should be noted:
134+
135+
##### Product images
136+
137+
The build action script `release.yml` builds all product images that defined in the `release.yaml` matrix section:
138+
139+
[source, yaml]
140+
----
141+
142+
name: Release product images
143+
on:
144+
push:
145+
tags:
146+
- '[0-9][0-9].[0-9]+.[0-9]+'
147+
148+
jobs:
149+
...
150+
strategy:
151+
fail-fast: false
152+
# If we want more parallelism we can schedule a dedicated task for every tuple (product, product version)
153+
matrix:
154+
product:
155+
# N.B. exclude base images!
156+
- airflow
157+
- zookeeper
158+
...
159+
----
160+
161+
Base images should be excluded from the build action as they need to be referenced by their manifest hashes in the product Dockerfiles and therefore should be built independently of the product images.
162+
163+
Also note that the tag pattern above is not using a regex (this functionality is not available for tag filtering) but uses glob-operators. The check is not totally watertight - we cannot for example enforce the "minor" version of the release to be limited to a digit between 1 and 12 - but this check is covered by the calling script `create-release-tag.sh`.
164+
165+
##### Operator images
166+
167+
Operator images are built by iterating over and pushing tags for the operator-repositories listed in the `operators` section of `config.yaml`:
168+
169+
[source, yaml]
170+
----
171+
images-repo: docker-images
172+
operators:
173+
- airflow
174+
- secret
175+
- commons
176+
- ...
177+
----

‎release/README.md

Lines changed: 0 additions & 67 deletions
This file was deleted.

‎release/config.yaml

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
---
2+
images-repo: docker-images
3+
operators:
4+
- airflow-operator
5+
- commons-operator
6+
- druid-operator
7+
- hbase-operator
8+
- hdfs-operator
9+
- hive-operator
10+
- kafka-operator
11+
- listener-operator
12+
- nifi-operator
13+
- opa-operator
14+
- trino-operator
15+
- secret-operator
16+
- spark-k8s-operator
17+
- superset-operator
18+
- zookeeper-operator

0 commit comments

Comments
 (0)