Skip to content

Commit 9dd0dfd

Browse files
romayalonromayalon
authored
Merge pull request #9033 from romayalon/romy-ci-warp-action
CI | Warp run action on Containerized and Non Containerized deployments
2 parents e5c8e86 + 46ef43e commit 9dd0dfd

12 files changed

+706
-21
lines changed

‎.github/workflows/warp-nc-tests.yaml

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
2+
name: Warp NC Tests
3+
on: [push, pull_request, workflow_dispatch]
4+
5+
jobs:
6+
warp-nc-tests:
7+
runs-on: ubuntu-latest
8+
timeout-minutes: 90
9+
concurrency:
10+
group: ${{ github.workflow }}-${{ github.ref }}
11+
cancel-in-progress: true
12+
steps:
13+
- name: Checkout noobaa-core
14+
uses: actions/checkout@v4
15+
with:
16+
repository: 'noobaa/noobaa-core'
17+
path: 'noobaa-core'
18+
19+
- name: Create Warp logs directory
20+
run: |
21+
set -x
22+
cd ./noobaa-core
23+
mkdir -p logs/warp-test-logs
24+
chmod 777 logs/warp-test-logs
25+
26+
- name: Run NC Warp tests
27+
run: |
28+
set -x
29+
cd ./noobaa-core
30+
make test-nc-warp
31+

‎.github/workflows/warp-tests.yaml

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
name: Warp Tests
2+
on: [push, pull_request, workflow_dispatch]
3+
4+
jobs:
5+
warp-tests:
6+
runs-on: ubuntu-latest
7+
timeout-minutes: 90
8+
concurrency:
9+
group: ${{ github.workflow }}-${{ github.ref }}
10+
cancel-in-progress: true
11+
steps:
12+
- name: Checkout noobaa-core
13+
uses: actions/checkout@v4
14+
with:
15+
repository: 'noobaa/noobaa-core'
16+
path: 'noobaa-core'
17+
18+
- name: Create Warp logs directory
19+
run: |
20+
set -x
21+
cd ./noobaa-core
22+
mkdir -p logs/warp-test-logs
23+
chmod 777 logs/warp-test-logs
24+
25+
- name: Run Warp tests
26+
run: |
27+
set -x
28+
cd ./noobaa-core
29+
make test-warp
30+

‎Makefile

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -337,6 +337,22 @@ test-cephs3: tester
337337
@$(call remove_docker_network)
338338
.PHONY: test-cephs3
339339

340+
test-warp: tester
341+
@echo "\033[1;34mRunning warp tests with Postgres.\033[0m"
342+
@$(call create_docker_network)
343+
@$(call run_postgres)
344+
@echo "\033[1;34mRunning warp tests\033[0m"
345+
$(CONTAINER_ENGINE) run $(CPUSET) --privileged --user root --network noobaa-net --name noobaa_$(GIT_COMMIT)_$(NAME_POSTFIX) --env "SUPPRESS_LOGS=$(SUPPRESS_LOGS)" --env "POSTGRES_HOST=coretest-postgres-$(GIT_COMMIT)-$(NAME_POSTFIX)" --env "POSTGRES_USER=noobaa" --env "DB_TYPE=postgres" --env "POSTGRES_DBNAME=coretest" -v $(PWD)/logs:/logs $(TESTER_TAG) "./src/test/system_tests/warp/run_warp_on_test_container.sh"
346+
@$(call stop_noobaa)
347+
@$(call stop_postgres)
348+
@$(call remove_docker_network)
349+
.PHONY: test-warp
350+
351+
test-nc-warp: tester
352+
@echo "\033[1;34mRunning warp tests on NC environment\033[0m"
353+
$(CONTAINER_ENGINE) run $(CPUSET) --privileged --user root --name noobaa_$(GIT_COMMIT)_$(NAME_POSTFIX) --env "SUPPRESS_LOGS=$(SUPPRESS_LOGS)" -v $(PWD)/logs:/logs $(TESTER_TAG) "./src/test/system_tests/warp/run_nc_warp_on_test_container.sh"
354+
.PHONY: test-nc-warp
355+
340356
test-nsfs-cephs3: tester
341357
@echo "\033[1;34mRunning Ceph S3 tests on NSFS Standalone platform\033[0m"
342358
$(CONTAINER_ENGINE) run $(CPUSET) --privileged --user root --name noobaa_$(GIT_COMMIT)_$(NAME_POSTFIX) --env "SUPPRESS_LOGS=$(SUPPRESS_LOGS)" -v $(PWD)/logs:/logs $(TESTER_TAG) "./src/test/system_tests/ceph_s3_tests/run_ceph_nsfs_test_on_test_container.sh"

‎docs/CI & Tests/warp.md

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
# Warp Github Action, Tests and Tool
2+
3+
1. [Introduction](#introduction)
4+
2. [Warp GitHub actions](#warp-github-actions)
5+
3. [Warp Makefile Targets](#warp-makefile-targets)
6+
4. [run_warp.js Tool](#run_warpjs-tool)
7+
5. [Manual Warp Installation](#manual-warp-installation)
8+
9+
10+
11+
## Introduction
12+
13+
[Warp](https://github.com/minio/warp) is a benchmarking tool for S3-compatible object storage systems, NooBaa CI runs Warp as performance/burst/scale tests for the NooBaa system on both containerized and Non Containerized flavors.
14+
Currently, Warp runs as part of our PR tests and in the future we will add long Warp runs as part of our nightly CI process.
15+
16+
## Warp GitHub actions
17+
18+
NooBaa CI contains 2 Github actions that build, configure and run Warp. These Github actions run automatically on every PR and on every push, and can run by workflow dispatch manually.
19+
* [Warp Tests](../../.github/workflows/warp-tests.yaml) - Based on NooBaa Tester image, runs Warp on standard NooBaa (db configuration).
20+
* [Warp NC Tests](../../.github/workflows/warp-nc-tests.yaml) - Based on NooBaa Tester image, runs Warp on non-containerized NooBaa (ConfigFS configuration).
21+
22+
Our next goal is to add longer Warp runs as part of NooBaa's nightly CI process.
23+
24+
## Warp Makefile Targets
25+
26+
One can run Warp tests on NooBaa using Warp Makefile targets -
27+
* `make run-warp` - Based on NooBaa Tester image, runs Warp on standard NooBaa (db configuration).
28+
* `make run-nc-warp` - Based on NooBaa Tester image, runs Warp on non-containerized NooBaa (ConfigFS configuration).
29+
30+
The above makefile targets, build NooBaa tester image, and later deploy NooBaa (DB/ConfigFS deployments), install warp, create default account and bucket and runs the run_warp.js tool.
31+
32+
## `run_warp.js` Tool
33+
34+
The `run_warp.js` script is designed to execute Warp performance tests for the NooBaa system. This script provides a streamlined way to configure and run Warp tests with various parameters.
35+
36+
37+
#### Usage
38+
39+
The script can be executed using Node.js. Below is the usage syntax:
40+
41+
```bash
42+
node run_warp.js [options]
43+
```
44+
45+
#### Command-Line Arguments
46+
47+
The following arguments are supported by the script:
48+
49+
| Argument | Description | Default Value |
50+
|---------------------|-----------------------------------------------------------------------------|--------------------------|
51+
| `--op` | Warp operation to run (`mix`, `put`, `get`, etc.). | `mixed` |
52+
| `--concurrency` | Number of workers to run the tests. | `5` (default workers) |
53+
| `--bucket` | Bucket name to run the tests on. | `warp-benchmark-bucket` |
54+
| `--duration` | Duration of the tests (e.g., `30s`, `1m`, `1h`). | `10m` |
55+
| `--disable-multipart` | Disable multipart upload (`true` or `false`). | `true` |
56+
| `--access-key` | Access key for the tests. | Derived from account. |
57+
| `--secret-key` | Secret key for the tests. | Derived from account. |
58+
| `--obj-size` | Object size for the tests (e.g., `1k`, `1m`, `1g`). | `1k` |
59+
| `--account-name` | Account name to use for the tests. | `warp_account` |
60+
| `--help` | Display usage information. | N/A |
61+
62+
63+
#### Warp Command Construction
64+
65+
The Warp command is constructed dynamically based on the provided arguments. Below is an example of the command:
66+
67+
```bash
68+
warp mixed --host=localhost:443 --access-key=<access_key> --secret-key=<secret_key> --bucket=<bucket> --obj.size=1k --duration=10m --disable-multipart=true --tls --insecure --concurrent 5
69+
```
70+
71+
#### Future Improvements
72+
73+
1. **Support for Additional Warp Features**:
74+
- Add support for more Warp operations and configurations.
75+
- Implement logging of test results in CSV format.
76+
77+
2. **Nightly CI Automation**:
78+
- Integrate the Warp tests into nightly CI/CD pipelines for automated performance testing.
79+
80+
---
81+
82+
83+
## Manual Warp Installation
84+
85+
Linux latest warp release - https://github.com/minio/warp/releases/download/v1.1.4/warp_Linux_x86_64.tar.gz
86+
87+
Darwin latest warp release - https://github.com/minio/warp/releases/download/v1.1.4/warp_Darwin_x86_64.tar.gz
88+
89+
```
90+
curl -L <warp_release_link> -o warp.tar.gz
91+
tar -xzf warp.tar.gz
92+
chmod +x warp
93+
mv warp /usr/local/bin/warp
94+
warp --version
95+
```

‎src/test/system_tests/ceph_s3_tests/test_ceph_nsfs_s3_config_setup.js

Lines changed: 1 addition & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,8 @@ dbg.set_process_name('test_ceph_s3');
1212

1313
const fs = require('fs');
1414
const os_utils = require('../../../util/os_utils');
15-
const test_utils = require('../../system_tests/test_utils');
16-
const { TYPES, ACTIONS } = require('../../../manage_nsfs/manage_nsfs_constants');
1715
const { CEPH_TEST } = require('./test_ceph_s3_constants.js');
16+
const { get_access_keys, create_account } = require('../nc_test_utils');
1817

1918
async function main() {
2019
try {
@@ -73,26 +72,7 @@ async function ceph_test_setup() {
7372
console.info('CEPH TEST CONFIGURATION: DONE');
7473
}
7574

76-
/**
77-
* get_access_keys returns account access keys using noobaa-cli
78-
* @param {string} account_name
79-
*/
80-
async function get_access_keys(account_name) {
81-
const options = { name: account_name, show_secrets: true };
82-
const res = await test_utils.exec_manage_cli(TYPES.ACCOUNT, ACTIONS.STATUS, options);
83-
const json_account = JSON.parse(res);
84-
const account_data = json_account.response.reply;
85-
return account_data.access_keys[0];
86-
}
8775

88-
/**
89-
* create_account creates accounts using noobaa-cli
90-
* @param {{ name?: string, uid?: number, gid?: number, anonymous?: boolean }} [options]
91-
*/
92-
async function create_account(options = {}) {
93-
const res = await test_utils.exec_manage_cli(TYPES.ACCOUNT, ACTIONS.ADD, options);
94-
console.log('Account Created', res);
95-
}
9676

9777
if (require.main === module) {
9878
main();

‎src/test/system_tests/nc_test_utils.js

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
/* Copyright (C) 2016 NooBaa */
2+
'use strict';
3+
4+
const test_utils = require('./test_utils');
5+
const { TYPES, ACTIONS } = require('../../manage_nsfs/manage_nsfs_constants');
6+
7+
/**
8+
* get_access_keys returns account access keys using noobaa-cli
9+
* @param {string} account_name
10+
*/
11+
async function get_access_keys(account_name) {
12+
const account_data = await get_account(account_name);
13+
return account_data.access_keys[0];
14+
}
15+
16+
/**
17+
* get_account returns account data using noobaa-cli
18+
* @param {string} account_name
19+
*/
20+
async function get_account(account_name) {
21+
const options = { name: account_name, show_secrets: true };
22+
const res = await test_utils.exec_manage_cli(TYPES.ACCOUNT, ACTIONS.STATUS, options);
23+
const json_account = JSON.parse(res);
24+
const account_data = json_account.response.reply;
25+
return account_data;
26+
}
27+
28+
/**
29+
* create_account creates account using noobaa-cli
30+
* @param {{ name?: string, uid?: number, gid?: number, anonymous?: boolean }} [options]
31+
*/
32+
async function create_account(options = {}) {
33+
const res = await test_utils.exec_manage_cli(TYPES.ACCOUNT, ACTIONS.ADD, options);
34+
console.log('NC Account Created', res);
35+
}
36+
37+
/**
38+
* create_bucket creates bucket using noobaa-cli
39+
* @param {{ name?: string, owner?: string, path?: string}} [options]
40+
*/
41+
async function create_bucket(options = {}) {
42+
const res = await test_utils.exec_manage_cli(TYPES.BUCKET, ACTIONS.ADD, options);
43+
console.log('NC Bucket Created', res);
44+
}
45+
46+
// EXPORTS
47+
exports.get_account = get_account;
48+
exports.get_access_keys = get_access_keys;
49+
exports.create_account = create_account;
50+
exports.create_bucket = create_bucket;

‎src/test/system_tests/warp/configure_warp.js

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
/* Copyright (C) 2016 NooBaa */
2+
"use strict";
3+
4+
const dbg = require('../../../util/debug_module')(__filename);
5+
dbg.set_process_name('test_warp_s3');
6+
7+
const { WARP_TEST } = require('./warp_constants.js');
8+
const { create_warp_account, create_warp_bucket } = require('./warp_utils.js');
9+
10+
async function main() {
11+
try {
12+
await warp_test_setup();
13+
} catch (err) {
14+
console.error(`Warp Setup Failed: ${err}`);
15+
process.exit(1);
16+
}
17+
process.exit(0);
18+
}
19+
20+
async function warp_test_setup() {
21+
console.info('WARP TEST CONFIGURATION:', JSON.stringify(WARP_TEST));
22+
await create_warp_account();
23+
await create_warp_bucket();
24+
console.info('WARP TEST SETUP DONE');
25+
}
26+
27+
if (require.main === module) {
28+
main();
29+
}
30+
31+

‎src/test/system_tests/warp/run_nc_warp_on_test_container.sh

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
#!/bin/bash
2+
3+
export PS4='\e[36m+ ${FUNCNAME:-main}\e[0m@\e[32m${BASH_SOURCE}:\e[35m${LINENO} \e[0m'
4+
5+
set -e
6+
# It will add to the logs every line that we run
7+
set -x
8+
9+
# ====================================================================================
10+
# Set the environment variables
11+
export email='admin@noobaa.io'
12+
export password=123456789
13+
14+
export ENDPOINT_PORT=6001
15+
export ENDPOINT_SSL_PORT=6443
16+
export S3_SERVICE_HOST=localhost
17+
18+
export CEPH_TEST_LOGS_DIR=/logs/warp-nc-test-logs
19+
export CONFIG_DIR=/etc/noobaa.conf.d/
20+
export FS_ROOT_1=/tmp/nsfs_root1/
21+
export FS_ROOT_2=/tmp/nsfs_root2/
22+
export WARP_BUCKET_PATH=${FS_ROOT_1}/warp-benchmark-bucket/
23+
export CONFIG_JS_allow_anonymous_access_in_test=true # Needed for allowing anon access for tests using ACL='public-read-write'
24+
25+
# ====================================================================================
26+
27+
# 1. Create configuration directory
28+
# 2. Create config.json file
29+
mkdir -p ${CONFIG_DIR}
30+
config='{"ALLOW_HTTP":true, "ENDPOINT_FORKS":2}'
31+
echo "$config" > ${CONFIG_DIR}/config.json
32+
33+
# 1. Create root directory for bucket creation
34+
# 2. Add permission to all users
35+
# this will allow the new accounts to create directories (buckets),
36+
# else we would see [Error: Permission denied] { code: 'EACCES' }
37+
mkdir -p ${FS_ROOT_1}
38+
mkdir -p ${FS_ROOT_2}
39+
mkdir -p ${WARP_BUCKET_PATH}
40+
chmod 777 ${FS_ROOT_1}
41+
chmod 777 ${FS_ROOT_2}
42+
chmod 777 ${WARP_BUCKET_PATH}
43+
44+
# Create the logs directory
45+
mkdir -p ${CEPH_TEST_LOGS_DIR}
46+
47+
48+
# ====================================================================================
49+
50+
# Install warp
51+
curl -L https://github.com/minio/warp/releases/download/v1.1.4/warp_Linux_x86_64.tar.gz -o warp_Linux_x86_64.tar.gz
52+
tar -xzf warp_Linux_x86_64.tar.gz
53+
chmod +x warp
54+
mv warp /usr/local/bin/warp
55+
warp --version
56+
57+
# ====================================================================================
58+
59+
# Deploy standalone NooBaa on the test container
60+
# And create the accounts needed for the Ceph tests
61+
./src/deploy/NVA_build/standalone_deploy_nsfs.sh
62+
63+
# ====================================================================================
64+
65+
cd /root/node_modules/noobaa-core/
66+
67+
# Configure the warp test
68+
node ./src/test/system_tests/warp/configure_warp.js
69+
70+
# ====================================================================================
71+
72+
# Run the warp tests
73+
node ./src/test/system_tests/warp/run_warp.js
74+
75+
# ====================================================================================

0 commit comments

Comments
 (0)