Skip to content

Commit 701c4e1

Browse files
laveryaemosbaughsgalsalehCraig O'Donnell
authored
feat: add support for HA registry (#655)
* feat: add support for ha registry * data replication factor 1 * f * fix * fix registry * f * update operator to v0.34.0 * create RBAC for migration job * fix rbac import * override operator image * operator v0.34.1 * f * f * fix * update migration rbac * fix * f * f * f * f * f * f * update operator version * update operator version * comment out ssh * add image override * fix role to allow secret creation * update operator image again * use released operator version * Add support for HA admin-console * custom eviction * no imagefs.inodesFree * prompt for high availability during node join (#657) * wip: add enable-ha cli flag * wait for k0s * check if HA is already enabled * fix reporting in join * addressing PR feedback * add unit tests * 60 backoff steps for HA migration wait * import WIP HA migration * three or more controller nodes * only wait for node and enable ha if joining controller node * fix controller check * clean up node join and ha messaging * bump ECO to 0.34.4 * no timeout for WaitForHAMigration * use node-role.kubernetes.io/control-plane=true selector * handle context cancelled in WaitForHAMigration * update enable HA prompt output * use time.After * revert time.After so check happens immediately * wait for controller node specifically * remove log * go mod tidy * use minimal ubuntu images * use lxd debian image * remove custom eviction * use debian 12 for proxy as well * wait for ha rqlite * only run ha test * install expect on node 2 * expand join command * fix --enable-flag position * fix message * fix message * not much to wait for in online mode * not much to wait for in online mode * wait for installation * unit test fixes * HA test changes * race condition testing * debian not ubuntu image * check HA migration status * fix unit tests, remove sleep * better operator * update operator * airgap ha test * include in tests * use right chart name for admin-console and docker-registry in operator * install curl for airgap debian nodes * e2e large runner * increase timeout * use released version of operator * another operator update * consolidate checks and increase timeout * fix app version * install curl 🤦 * enable all tests * more test fixes * fix syntax * enable ssh * keep cluster around... * add appver- prefix --------- Co-authored-by: Ethan Mosbaugh <ethan@replicated.com> Co-authored-by: Salah Al Saleh <sg.alsaleh@gmail.com> Co-authored-by: Craig O'Donnell <craig@replicated.com>
1 parent d15a196 commit 701c4e1

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+1507
-179
lines changed

‎.github/actions/e2e/action.yml

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,10 @@ runs:
7878
sudo apt-get update -y
7979
sudo apt install apparmor-utils -y
8080
sudo aa-complain /etc/apparmor.d/*
81+
- name: Setup Go
82+
uses: actions/setup-go@v5
83+
with:
84+
go-version-file: go.mod
8185
- name: E2E
8286
shell: bash
8387
run: |
@@ -133,4 +137,7 @@ runs:
133137
shell: bash
134138
# - name: Setup upterm session (ssh)
135139
# uses: lhotari/action-upterm@v1
136-
# if: always()
140+
# if: failure()
141+
# with:
142+
# ## If no one connects after 5 minutes, shut down server.
143+
# wait-timeout-minutes: 5

‎.github/workflows/dependencies.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ jobs:
1414
- name: Set up Go
1515
uses: actions/setup-go@v5
1616
with:
17-
go-version: "1.21"
17+
go-version-file: go.mod
1818
- name: AdminConsole
1919
run: |
2020
export VERSION=`curl https://api.github.com/repos/replicatedhq/kots-helm/tags | jq -r .[].name | grep -v alpha | head -1 | tr -d v`

‎.github/workflows/pull-request.yaml

Lines changed: 13 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ jobs:
1414
- name: Setup Go
1515
uses: actions/setup-go@v5
1616
with:
17-
go-version: "1.21.0"
17+
go-version-file: go.mod
1818
- name: Go vet
1919
run: |
2020
make vet
@@ -31,7 +31,7 @@ jobs:
3131
- name: Setup Go
3232
uses: actions/setup-go@v5
3333
with:
34-
go-version: "1.21.0"
34+
go-version-file: go.mod
3535
- name: Go vet
3636
run: |
3737
make vet
@@ -49,7 +49,7 @@ jobs:
4949
- name: Setup Go
5050
uses: actions/setup-go@v5
5151
with:
52-
go-version: "1.21.0"
52+
go-version-file: go.mod
5353
- name: Build Linux AMD64 and Output Metadata
5454
run: |
5555
export SHORT_SHA=dev-$(git rev-parse --short=7 HEAD)
@@ -160,13 +160,13 @@ jobs:
160160
output/bin/embedded-cluster-release-builder
161161
162162
e2e:
163-
runs-on: ubuntu-latest
163+
runs-on: ${{ matrix.runner || 'ubuntu-latest' }}
164164
needs:
165165
- build
166166
strategy:
167167
fail-fast: false
168168
matrix:
169-
tests:
169+
test:
170170
- TestSingleNodeInstallation
171171
- TestSingleNodeInstallationAlmaLinux8
172172
- TestSingleNodeInstallationDebian12
@@ -185,13 +185,18 @@ jobs:
185185
- TestOldVersionUpgrade
186186
- TestMaterialize
187187
- TestLocalArtifactMirror
188-
- TestSingleNodeAirgapUpgradeUbuntuJammy
188+
- TestSingleNodeAirgapUpgrade
189189
- TestInstallSnapshotFromReplicatedApp
190-
- TestMultiNodeAirgapUpgradeUbuntuJammy
190+
- TestMultiNodeAirgapUpgrade
191191
- TestSingleNodeDisasterRecovery
192192
- TestSingleNodeResumeDisasterRecovery
193193
- TestSingleNodeAirgapDisasterRecovery
194+
- TestMultiNodeHAInstallation
195+
- TestMultiNodeAirgapHAInstallation
194196
- TestMultiNodeAirgapUpgradeSameK0s
197+
include:
198+
- test: TestMultiNodeAirgapHAInstallation
199+
runner: embedded-cluster
195200
steps:
196201
- name: Checkout
197202
uses: actions/checkout@v4
@@ -203,7 +208,7 @@ jobs:
203208

204209
- uses: ./.github/actions/e2e
205210
with:
206-
test-name: '${{ matrix.tests }}'
211+
test-name: '${{ matrix.test }}'
207212
airgap-license-id: ${{ secrets.STAGING_EMBEDDED_CLUSTER_AIRGAP_LICENSE_ID }}
208213
snapshot-license-id: ${{ secrets.STAGING_EMBEDDED_CLUSTER_SNAPSHOT_LICENSE_ID }}
209214
snapshot-license: ${{ secrets.STAGING_EMBEDDED_CLUSTER_SNAPSHOT_LICENSE }}

‎.github/workflows/release-dev.yaml

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ jobs:
1616
- name: Set up Go
1717
uses: actions/setup-go@v5
1818
with:
19-
go-version: "1.21"
19+
go-version-file: go.mod
2020
- name: Build Linux AMD64 and Output Metadata
2121
run: |
2222
export SHORT_SHA=dev-$(git rev-parse --short=7 HEAD)
@@ -111,13 +111,13 @@ jobs:
111111
output/bin/embedded-cluster-previous-k0s
112112
output/bin/embedded-cluster-release-builder
113113
e2e:
114-
runs-on: ubuntu-latest
114+
runs-on: ${{ matrix.runner || 'ubuntu-latest' }}
115115
needs:
116116
- releaser
117117
strategy:
118118
fail-fast: false
119119
matrix:
120-
tests:
120+
test:
121121
- TestSingleNodeInstallation
122122
- TestSingleNodeInstallationAlmaLinux8
123123
- TestSingleNodeInstallationDebian12
@@ -136,13 +136,18 @@ jobs:
136136
- TestOldVersionUpgrade
137137
- TestMaterialize
138138
- TestLocalArtifactMirror
139-
- TestSingleNodeAirgapUpgradeUbuntuJammy
139+
- TestSingleNodeAirgapUpgrade
140140
- TestInstallSnapshotFromReplicatedApp
141-
- TestMultiNodeAirgapUpgradeUbuntuJammy
141+
- TestMultiNodeAirgapUpgrade
142142
- TestSingleNodeDisasterRecovery
143143
- TestSingleNodeResumeDisasterRecovery
144144
- TestSingleNodeAirgapDisasterRecovery
145+
- TestMultiNodeHAInstallation
146+
- TestMultiNodeAirgapHAInstallation
145147
- TestMultiNodeAirgapUpgradeSameK0s
148+
include:
149+
- test: TestMultiNodeAirgapHAInstallation
150+
runner: embedded-cluster
146151
steps:
147152
- name: Checkout
148153
uses: actions/checkout@v4
@@ -153,7 +158,7 @@ jobs:
153158
path: output/bin
154159
- uses: ./.github/actions/e2e
155160
with:
156-
test-name: '${{ matrix.tests }}'
161+
test-name: '${{ matrix.test }}'
157162
airgap-license-id: ${{ secrets.STAGING_EMBEDDED_CLUSTER_AIRGAP_LICENSE_ID }}
158163
snapshot-license-id: ${{ secrets.STAGING_EMBEDDED_CLUSTER_SNAPSHOT_LICENSE_ID }}
159164
snapshot-license: ${{ secrets.STAGING_EMBEDDED_CLUSTER_SNAPSHOT_LICENSE }}

‎.github/workflows/release-prod.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ jobs:
1818
- name: Set up Go
1919
uses: actions/setup-go@v5
2020
with:
21-
go-version: "1.21"
21+
go-version-file: go.mod
2222
- name: Build linux-amd64
2323
run: |
2424
make embedded-cluster-linux-amd64 VERSION=$TAG_NAME

‎Makefile

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,18 +4,21 @@ ARCH := $(shell uname -m)
44
APP_NAME = embedded-cluster
55
ADMIN_CONSOLE_CHART_URL = oci://registry.replicated.com/library
66
ADMIN_CONSOLE_CHART_NAME = admin-console
7-
ADMIN_CONSOLE_CHART_VERSION = 1.109.8
7+
ADMIN_CONSOLE_CHART_VERSION = 1.109.9-build.1
88
ADMIN_CONSOLE_IMAGE_OVERRIDE =
99
ADMIN_CONSOLE_MIGRATIONS_IMAGE_OVERRIDE =
1010
EMBEDDED_OPERATOR_CHART_URL = oci://registry.replicated.com/library
1111
EMBEDDED_OPERATOR_CHART_NAME = embedded-cluster-operator
12-
EMBEDDED_OPERATOR_CHART_VERSION = 0.33.2
12+
EMBEDDED_OPERATOR_CHART_VERSION = 0.34.6
1313
EMBEDDED_OPERATOR_UTILS_IMAGE = busybox:1.36.1
1414
EMBEDDED_CLUSTER_OPERATOR_IMAGE_OVERRIDE =
1515
OPENEBS_CHART_URL = https://openebs.github.io/openebs
1616
OPENEBS_CHART_NAME = openebs/openebs
1717
OPENEBS_CHART_VERSION = 4.0.1
1818
OPENEBS_UTILS_VERSION = 4.0.0
19+
SEAWEEDFS_CHART_URL = https://seaweedfs.github.io/seaweedfs/helm
20+
SEAWEEDFS_CHART_NAME = seaweedfs/seaweedfs
21+
SEAWEEDFS_CHART_VERSION = 3.67.0
1922
REGISTRY_CHART_URL = https://helm.twun.io
2023
REGISTRY_CHART_NAME = twuni/docker-registry
2124
REGISTRY_CHART_VERSION = 2.2.3
@@ -51,6 +54,9 @@ LD_FLAGS = -X github.com/replicatedhq/embedded-cluster/pkg/defaults.K0sVersion=$
5154
-X github.com/replicatedhq/embedded-cluster/pkg/addons/openebs.ChartName=$(OPENEBS_CHART_NAME) \
5255
-X github.com/replicatedhq/embedded-cluster/pkg/addons/openebs.Version=$(OPENEBS_CHART_VERSION) \
5356
-X github.com/replicatedhq/embedded-cluster/pkg/addons/openebs.UtilsVersion=$(OPENEBS_UTILS_VERSION) \
57+
-X github.com/replicatedhq/embedded-cluster/pkg/addons/seaweedfs.ChartURL=$(SEAWEEDFS_CHART_URL) \
58+
-X github.com/replicatedhq/embedded-cluster/pkg/addons/seaweedfs.ChartName=$(SEAWEEDFS_CHART_NAME) \
59+
-X github.com/replicatedhq/embedded-cluster/pkg/addons/seaweedfs.Version=$(SEAWEEDFS_CHART_VERSION) \
5460
-X github.com/replicatedhq/embedded-cluster/pkg/addons/registry.ChartURL=$(REGISTRY_CHART_URL) \
5561
-X github.com/replicatedhq/embedded-cluster/pkg/addons/registry.ChartName=$(REGISTRY_CHART_NAME) \
5662
-X github.com/replicatedhq/embedded-cluster/pkg/addons/registry.Version=$(REGISTRY_CHART_VERSION) \
@@ -147,7 +153,7 @@ embedded-cluster-darwin-arm64: go.mod
147153

148154
.PHONY: unit-tests
149155
unit-tests:
150-
go test -v $(shell go list ./... | grep -v /e2e)
156+
go test -v ./pkg/... ./cmd/...
151157

152158
.PHONY: vet
153159
vet: static

0 commit comments

Comments
 (0)