Skip to content

Commit b83cb6e

Browse files
XuanwoXuanwo
authored
feat: Add bendsave which can backup and restore databend data (#17503)
* squash commits Signed-off-by: Xuanwo <github@xuanwo.io> * Fix force load Signed-off-by: Xuanwo <github@xuanwo.io> * backup works! Signed-off-by: Xuanwo <github@xuanwo.io> * Fully test Signed-off-by: Xuanwo <github@xuanwo.io> * Fix test Signed-off-by: Xuanwo <github@xuanwo.io> * Fix actions Signed-off-by: Xuanwo <github@xuanwo.io> * Try fix ci Signed-off-by: Xuanwo <github@xuanwo.io> * Fix insert in epochfs Signed-off-by: Xuanwo <github@xuanwo.io> * allow more time Signed-off-by: Xuanwo <github@xuanwo.io> * Add readme Signed-off-by: Xuanwo <github@xuanwo.io> * fix typo Signed-off-by: Xuanwo <github@xuanwo.io> * Update cargo.lock Signed-off-by: Xuanwo <github@xuanwo.io> * remove unneeded changes Signed-off-by: Xuanwo <github@xuanwo.io> * Add license check to backup tool bendsave * Replace epochfs with new backup implementation The commit switches from using epochfs to a new direct storage-to-storage backup implementation for bendsave, removing the checkpoint-based approach. The main changes: - Remove epochfs dependency and checkpoint-based backup/restore - Add direct storage copy util function - Update command line interface to remove checkpoint flag - Rename storage functions for clarity * Remove checkpoint flag from bendsave restore * Fix typo Signed-off-by: Xuanwo <github@xuanwo.io> * Avoid check license while restore Signed-off-by: Xuanwo <github@xuanwo.io> * Init query first Signed-off-by: Xuanwo <github@xuanwo.io> * Fix build Signed-off-by: Xuanwo <github@xuanwo.io> * Fix init query Signed-off-by: Xuanwo <github@xuanwo.io> * Fix clippy Signed-off-by: Xuanwo <github@xuanwo.io> --------- Signed-off-by: Xuanwo <github@xuanwo.io>
1 parent a948fba commit b83cb6e

File tree

24 files changed

+1129
-86
lines changed

24 files changed

+1129
-86
lines changed

‎.github/actions/build_linux/action.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@ runs:
6868
if: inputs.artifacts == 'all'
6969
shell: bash
7070
run: |
71-
artifacts="meta,metactl,metabench,query,sqllogictests"
71+
artifacts="meta,metactl,metabench,query,sqllogictests,bendsave"
7272
binaries=""
7373
for artifact in ${artifacts//,/ }; do
7474
binaries="${binaries} --bin databend-$artifact"

‎.github/actions/test_sqllogic_standalone_linux_minio_with_bendsave/action.yml

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
name: "Test sqllogic Standalone(minio) with bendsave"
2+
description: "Running sqllogic tests in standalone mode"
3+
4+
runs:
5+
using: "composite"
6+
steps:
7+
- uses: ./.github/actions/setup_test
8+
with:
9+
artifacts: sqllogictests,meta,query,bendsave
10+
11+
- name: Minio Setup for (ubuntu-latest only)
12+
shell: bash
13+
run: |
14+
docker run -d --network host --name minio \
15+
-e "MINIO_ACCESS_KEY=minioadmin" \
16+
-e "MINIO_SECRET_KEY=minioadmin" \
17+
-e "MINIO_ADDRESS=:9900" \
18+
-v /tmp/data:/data \
19+
-v /tmp/config:/root/.minio \
20+
minio/minio server /data
21+
22+
export AWS_ACCESS_KEY_ID=minioadmin
23+
export AWS_SECRET_ACCESS_KEY=minioadmin
24+
export AWS_EC2_METADATA_DISABLED=true
25+
26+
aws --endpoint-url http://127.0.0.1:9900/ s3 mb s3://testbucket
27+
aws --endpoint-url http://127.0.0.1:9900/ s3 mb s3://backupbucket
28+
29+
- name: Run sqllogic Tests with Standalone mode
30+
shell: bash
31+
env:
32+
TEST_HANDLERS: "http"
33+
run: bash ./scripts/ci/ci-run-sqllogic-tests-minio-with-bendsave.sh tpch
34+
35+
- name: Run bendsave backup
36+
shell: bash
37+
env:
38+
AWS_ACCESS_KEY_ID: minioadmin
39+
AWS_SECRET_ACCESS_KEY: minioadmin
40+
AWS_EC2_METADATA_DISABLED: true
41+
AWS_REGION: us-west-2
42+
run: |
43+
export STORAGE_TYPE=s3
44+
export STORAGE_S3_BUCKET=testbucket
45+
export STORAGE_S3_ROOT=admin
46+
export STORAGE_S3_ENDPOINT_URL=http://127.0.0.1:9900
47+
export STORAGE_S3_ACCESS_KEY_ID=minioadmin
48+
export STORAGE_S3_SECRET_ACCESS_KEY=minioadmin
49+
export STORAGE_ALLOW_INSECURE=true
50+
51+
./target/${{ env.BUILD_PROFILE }}/databend-bendsave backup --from ./scripts/ci/deploy/config/databend-query-node-1.toml --to s3://backupbucket?endpoint=http://127.0.0.1:9900/
52+
53+
aws --endpoint-url http://127.0.0.1:9900/ s3 ls s3://backupbucket --recursive
54+
55+
- name: Destroy the existing services.
56+
shell: bash
57+
env:
58+
AWS_ACCESS_KEY_ID: minioadmin
59+
AWS_SECRET_ACCESS_KEY: minioadmin
60+
AWS_EC2_METADATA_DISABLED: true
61+
AWS_REGION: us-west-2
62+
run: |
63+
# kill all services
64+
pkill databend-query
65+
pkill databend-meta
66+
# destory databend query
67+
aws --endpoint-url http://127.0.0.1:9900/ s3 rm s3://testbucket --recursive
68+
# destory databend meta
69+
rm -rf /tmp/databend/meta1
70+
71+
- name: Run bendsave restore
72+
shell: bash
73+
env:
74+
AWS_ACCESS_KEY_ID: minioadmin
75+
AWS_SECRET_ACCESS_KEY: minioadmin
76+
AWS_EC2_METADATA_DISABLED: true
77+
AWS_REGION: us-west-2
78+
run: |
79+
export STORAGE_TYPE=s3
80+
export STORAGE_S3_BUCKET=testbucket
81+
export STORAGE_S3_ROOT=admin
82+
export STORAGE_S3_ENDPOINT_URL=http://127.0.0.1:9900
83+
export STORAGE_S3_ACCESS_KEY_ID=minioadmin
84+
export STORAGE_S3_SECRET_ACCESS_KEY=minioadmin
85+
export STORAGE_ALLOW_INSECURE=true
86+
87+
./target/${{ env.BUILD_PROFILE }}/databend-bendsave restore --from s3://backupbucket?endpoint=http://127.0.0.1:9900/ --to-query ./scripts/ci/deploy/config/databend-query-node-1.toml --to-meta ./scripts/ci/deploy/config/databend-meta-node-for-bendsave.toml --confirm
88+
89+
- name: Run sqllogic Tests with Standalone mode again for testing
90+
shell: bash
91+
env:
92+
TEST_HANDLERS: "http"
93+
run: bash ./scripts/ci/ci-run-sqllogic-tests-minio-with-bendsave.sh tpch

‎.github/workflows/reuse.sqllogic.yml

Lines changed: 24 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ env:
2525

2626
jobs:
2727
management_mode:
28-
runs-on: [self-hosted, X64, Linux, 2c8g, "${{ inputs.runner_provider }}"]
28+
runs-on: [ self-hosted, X64, Linux, 2c8g, "${{ inputs.runner_provider }}" ]
2929
steps:
3030
- uses: actions/checkout@v4
3131
- uses: ./.github/actions/test_sqllogic_management_mode_linux
@@ -72,7 +72,7 @@ jobs:
7272
name: test-sqllogic-standalone-${{ matrix.tests.dirs }}-${{ matrix.handler }}
7373

7474
standalone_udf_server:
75-
runs-on: [self-hosted, X64, Linux, 2c8g, "${{ inputs.runner_provider }}"]
75+
runs-on: [ self-hosted, X64, Linux, 2c8g, "${{ inputs.runner_provider }}" ]
7676
steps:
7777
- uses: actions/checkout@v4
7878
- name: Start UDF Server
@@ -94,7 +94,7 @@ jobs:
9494
name: test-sqllogic-standalone-udf-server
9595

9696
standalone_cloud:
97-
runs-on: [self-hosted, X64, Linux, 4c16g, "${{ inputs.runner_provider }}"]
97+
runs-on: [ self-hosted, X64, Linux, 4c16g, "${{ inputs.runner_provider }}" ]
9898
steps:
9999
- uses: actions/checkout@v4
100100
- name: Start Cloud Control Server
@@ -115,7 +115,7 @@ jobs:
115115
name: test-sqllogic-standalone-cloud
116116

117117
standalone_minio:
118-
runs-on: [self-hosted, X64, Linux, 2c8g, "${{ inputs.runner_provider }}"]
118+
runs-on: [ self-hosted, X64, Linux, 2c8g, "${{ inputs.runner_provider }}" ]
119119
strategy:
120120
fail-fast: false
121121
matrix:
@@ -141,8 +141,24 @@ jobs:
141141
with:
142142
name: test-sqllogic-standalone-minio-${{ matrix.dirs }}-${{ matrix.handler }}-${{ matrix.format }}
143143

144+
standalone_minio_with_bendsave:
145+
runs-on: [ self-hosted, X64, Linux, 2c8g, "${{ inputs.runner_provider }}" ]
146+
steps:
147+
- uses: actions/checkout@v4
148+
- uses: ./.github/actions/setup_license
149+
with:
150+
runner_provider: ${{ inputs.runner_provider }}
151+
type: ${{ inputs.license_type }}
152+
- uses: ./.github/actions/test_sqllogic_standalone_linux_minio_with_bendsave
153+
timeout-minutes: 30
154+
- name: Upload failure
155+
if: failure()
156+
uses: ./.github/actions/artifact_failure
157+
with:
158+
name: test-sqllogic-standalone-minio-tpch-http-with-bendsave
159+
144160
standalone_iceberg_tpch:
145-
runs-on: [self-hosted, X64, Linux, 4c16g, "${{ inputs.runner_provider }}"]
161+
runs-on: [ self-hosted, X64, Linux, 4c16g, "${{ inputs.runner_provider }}" ]
146162
steps:
147163
- uses: actions/checkout@v4
148164
- uses: actions/setup-java@v4
@@ -201,7 +217,7 @@ jobs:
201217
name: test-sqllogic-cluster-${{ matrix.tests.dirs }}-${{ matrix.handler }}
202218

203219
stage:
204-
runs-on: [self-hosted, X64, Linux, 2c8g, "${{ inputs.runner_provider }}"]
220+
runs-on: [ self-hosted, X64, Linux, 2c8g, "${{ inputs.runner_provider }}" ]
205221
strategy:
206222
fail-fast: false
207223
matrix:
@@ -230,7 +246,7 @@ jobs:
230246
name: test-sqllogic-stage-${{ matrix.storage }}-${{ matrix.handler }}-${{ matrix.dedup }}
231247

232248
standalone_no_table_meta_cache:
233-
runs-on: [self-hosted, X64, Linux, 2c8g, "${{ inputs.runner_provider }}"]
249+
runs-on: [ self-hosted, X64, Linux, 2c8g, "${{ inputs.runner_provider }}" ]
234250
strategy:
235251
fail-fast: false
236252
matrix:
@@ -254,7 +270,7 @@ jobs:
254270
name: test-sqllogic-standalone-no-table-meta-cache-${{ matrix.dirs }}-${{ matrix.handler }}
255271

256272
ee:
257-
runs-on: [self-hosted, X64, Linux, 2c8g, "${{ inputs.runner_provider }}"]
273+
runs-on: [ self-hosted, X64, Linux, 2c8g, "${{ inputs.runner_provider }}" ]
258274
strategy:
259275
fail-fast: false
260276
matrix:

0 commit comments

Comments
 (0)