Skip to content

Commit 2833a37

Browse files
mkolasinski-splunkmkolasinski-splunk
authored
chore(release): merge develop to main (#279)
Test runs: https://github.com/splunk/splunk-add-on-for-amazon-web-services/actions/runs/9269557494 https://github.com/splunk/test-addonfactory-repo/actions/runs/9270329208 Changes possibly affecting customers: - add ui_marker to enable splitting UI tests execution - fix for failing upload-artifact@v4 due to multiple artifacts with same name - replace edplato/trufflehog-actions-scan action with official trufflehog action latest release - bump version of workflow-engine-base image to v4.0 (Ubuntu 20 -> Ubuntu 22) Changes not affecting customers: - wfe-test-runner-action update to v5 and change of the way that browser is determined in k8s - docs and ci related updates Also note that starting from release v4.16.0 some assets are removed from TA releases: 1. installation-actions.json 2. installation-update.json 3. \<TA-release\>-_forwarders.spl 4. \<TA-release\>-_indexers.spl 5. \<TA-release\>-_search_heads.spl
2 parents 70fa943 + 9543022 commit 2833a37

File tree

5 files changed

+83
-46
lines changed

5 files changed

+83
-46
lines changed

‎.github/workflows/build-test-release.yaml

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -58,9 +58,14 @@ jobs:
5858
update-semver:
5959
if: startsWith(github.ref, 'refs/tags/v')
6060
needs: publish
61-
permissions:
62-
contents: write
6361
runs-on: ubuntu-latest
6462
steps:
6563
- uses: actions/checkout@v4
66-
- uses: haya14busa/action-update-semver@v1
64+
- uses: splunk/addonfactory-update-semver@v1
65+
env:
66+
GITHUB_TOKEN: ${{ secrets.GH_TOKEN_ADMIN }}
67+
with:
68+
git_committer_name: ${{ secrets.SA_GH_USER_NAME }}
69+
git_committer_email: ${{ secrets.SA_GH_USER_EMAIL }}
70+
gpg_private_key: ${{ secrets.SA_GPG_PRIVATE_KEY }}
71+
passphrase: ${{ secrets.SA_GPG_PASSPHRASE }}

‎.github/workflows/reusable-build-test-release.yml

Lines changed: 48 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,13 @@ on:
44
inputs:
55
marker:
66
required: false
7-
description: 'Parallel run marker'
7+
description: 'Parallel run mod_input marker'
8+
type: string
9+
default: >-
10+
[""]
11+
ui_marker:
12+
required: false
13+
description: 'Parallel run ui marker'
814
type: string
915
default: >-
1016
[""]
@@ -17,7 +23,7 @@ on:
1723
required: false
1824
description: "branch for k8s manifests to run the tests on"
1925
type: string
20-
default: "v2.0.0"
26+
default: "v3.0.0"
2127
secrets:
2228
GH_TOKEN_ADMIN:
2329
description: Github admin token
@@ -258,11 +264,12 @@ jobs:
258264
submodules: false
259265
fetch-depth: "0"
260266
ref: ${{ github.head_ref }}
261-
- name: Trufflehog Actions Scan
262-
uses: edplato/trufflehog-actions-scan@v0.9l-beta
267+
- name: Secret Scanning Trufflehog
268+
uses: trufflesecurity/trufflehog@v3.77.0
263269
with:
264-
scanArguments: "--max_dept 5 -x .github/workflows/exclude-patterns.txt --allow .github/workflows/trufflehog-false-positive.json"
265-
270+
extra_args: -x .github/workflows/exclude-patterns.txt --json
271+
version: 3.77.0
272+
266273
semgrep:
267274
runs-on: ubuntu-latest
268275
name: security-sast-semgrep
@@ -904,7 +911,7 @@ jobs:
904911
sc4s: ${{ fromJson(needs.meta.outputs.matrix_supportedSC4S) }}
905912

906913
container:
907-
image: ghcr.io/splunk/workflow-engine-base:3.0.0
914+
image: ghcr.io/splunk/workflow-engine-base:4.0.0
908915
env:
909916
ARGO_SERVER: ${{ needs.setup.outputs.argo-server }}
910917
ARGO_HTTP1: ${{ needs.setup.outputs.argo-http1 }}
@@ -963,7 +970,7 @@ jobs:
963970
continue-on-error: true
964971
env:
965972
ARGO_TOKEN: ${{ steps.get-argo-token.outputs.argo-token }}
966-
uses: splunk/wfe-test-runner-action@v3.0
973+
uses: splunk/wfe-test-runner-action@v5.0
967974
with:
968975
splunk: ${{ matrix.splunk.version }}${{ secrets.OTHER_TA_REQUIRED_CONFIGS }}
969976
test-type: ${{ env.TEST_TYPE }}
@@ -1137,7 +1144,7 @@ jobs:
11371144
sc4s: ${{ fromJson(needs.meta.outputs.matrix_supportedSC4S) }}
11381145

11391146
container:
1140-
image: ghcr.io/splunk/workflow-engine-base:3.0.0
1147+
image: ghcr.io/splunk/workflow-engine-base:4.0.0
11411148
env:
11421149
ARGO_SERVER: ${{ needs.setup.outputs.argo-server }}
11431150
ARGO_HTTP1: ${{ needs.setup.outputs.argo-http1 }}
@@ -1195,7 +1202,7 @@ jobs:
11951202
continue-on-error: true
11961203
env:
11971204
ARGO_TOKEN: ${{ steps.get-argo-token.outputs.argo-token }}
1198-
uses: splunk/wfe-test-runner-action@v3.0
1205+
uses: splunk/wfe-test-runner-action@v5.0
11991206
with:
12001207
splunk: ${{ matrix.splunk.version }}
12011208
test-type: ${{ env.TEST_TYPE }}
@@ -1347,9 +1354,9 @@ jobs:
13471354
splunk: ${{ fromJson(needs.meta.outputs.matrix_combinedSplunkversion) }}
13481355
browser: [ "chrome" ]
13491356
vendor-version: ${{ fromJson(needs.meta.outputs.matrix_supportedUIVendors) }}
1350-
1357+
marker: ${{ fromJson(inputs.ui_marker) }}
13511358
container:
1352-
image: ghcr.io/splunk/workflow-engine-base:3.0.0
1359+
image: ghcr.io/splunk/workflow-engine-base:4.0.0
13531360
env:
13541361
ARGO_SERVER: ${{ needs.setup.outputs.argo-server }}
13551362
ARGO_HTTP1: ${{ needs.setup.outputs.argo-http1 }}
@@ -1359,6 +1366,7 @@ jobs:
13591366
SPLUNK_VERSION_BASE: ${{ matrix.splunk.version }}${{ secrets.OTHER_TA_REQUIRED_CONFIGS }}
13601367
TEST_TYPE: "ui"
13611368
TEST_ARGS: "--browser ${{ matrix.browser }}"
1369+
TEST_BROWSER: ${{ matrix.browser }}
13621370
permissions:
13631371
actions: read
13641372
deployments: read
@@ -1402,17 +1410,29 @@ jobs:
14021410
JOB_NAME=${JOB_NAME//[_.:]/-}
14031411
JOB_NAME=$(echo "$JOB_NAME" | tr '[:upper:]' '[:lower:]')
14041412
echo "job-name=$JOB_NAME" >> "$GITHUB_OUTPUT"
1413+
- name: create test argument
1414+
id: create-test-arg
1415+
shell: bash
1416+
run: |
1417+
TEST_ARG_M=""
1418+
EMPTY_MARKER="[]"
1419+
1420+
if [[ "${{ inputs.ui_marker }}" != "$EMPTY_MARKER" ]]; then
1421+
TEST_ARG_M="-m"
1422+
fi
1423+
1424+
echo "test-arg=$TEST_ARG_M" >> "$GITHUB_OUTPUT"
14051425
- name: run-tests
14061426
id: run-tests
14071427
timeout-minutes: 340
14081428
continue-on-error: true
14091429
env:
14101430
ARGO_TOKEN: ${{ steps.get-argo-token.outputs.argo-token }}
1411-
uses: splunk/wfe-test-runner-action@v3.0
1431+
uses: splunk/wfe-test-runner-action@v5.0
14121432
with:
14131433
splunk: ${{ matrix.splunk.version }}${{ secrets.OTHER_TA_REQUIRED_CONFIGS }}
14141434
test-type: ${{ env.TEST_TYPE }}
1415-
test-args: ${{ env.TEST_ARGS }}
1435+
test-args: ${{ env.TEST_ARGS }} ${{ steps.create-test-arg.outputs.test-arg }} ${{ matrix.marker }}
14161436
job-name: ${{ steps.create-job-name.outputs.job-name }}
14171437
labels: ${{ needs.setup.outputs.labels }}
14181438
workflow-tmpl-name: ${{ needs.setup.outputs.argo-workflow-tmpl-name }}
@@ -1422,6 +1442,7 @@ jobs:
14221442
vendor-version: ${{ matrix.vendor-version.image }}
14231443
sc4s-version: "No"
14241444
k8s-manifests-branch: ${{ needs.setup.outputs.k8s-manifests-branch }}
1445+
test-browser: ${{ env.TEST_BROWSER }}
14251446
- name: Read secrets from AWS Secrets Manager again into environment variables in case credential rotation
14261447
id: update-argo-token
14271448
if: ${{ !cancelled() }}
@@ -1522,13 +1543,13 @@ jobs:
15221543
- uses: actions/upload-artifact@v4
15231544
if: ${{ !cancelled() }}
15241545
with:
1525-
name: archive splunk ${{ matrix.splunk.version }}${{ secrets.OTHER_TA_REQUIRED_CONFIGS }} ${{ env.TEST_TYPE }} ${{ matrix.browser }} ${{ matrix.vendor-version.image }} tests artifacts
1546+
name: archive splunk ${{ matrix.splunk.version }}${{ secrets.OTHER_TA_REQUIRED_CONFIGS }} ${{ env.TEST_TYPE }} ${{ matrix.browser }} ${{ matrix.vendor-version.image }} ${{ matrix.marker }} tests artifacts
15261547
path: |
15271548
${{ needs.setup.outputs.directory-path }}/test-results
15281549
- uses: actions/upload-artifact@v4
15291550
if: ${{ !cancelled() }}
15301551
with:
1531-
name: archive splunk ${{ matrix.splunk.version }}${{ secrets.OTHER_TA_REQUIRED_CONFIGS }} ${{ env.TEST_TYPE }} ${{ matrix.browser }} ${{ matrix.vendor-version.image }} tests logs
1552+
name: archive splunk ${{ matrix.splunk.version }}${{ secrets.OTHER_TA_REQUIRED_CONFIGS }} ${{ env.TEST_TYPE }} ${{ matrix.browser }} ${{ matrix.vendor-version.image }} ${{ matrix.marker }} tests logs
15321553
path: |
15331554
${{ needs.setup.outputs.directory-path }}/argo-logs
15341555
- name: Test Report
@@ -1568,7 +1589,7 @@ jobs:
15681589
vendor-version: ${{ fromJson(needs.meta.outputs.matrix_supportedModinputFunctionalVendors) }}
15691590
marker: ${{ fromJson(inputs.marker) }}
15701591
container:
1571-
image: ghcr.io/splunk/workflow-engine-base:3.0.0
1592+
image: ghcr.io/splunk/workflow-engine-base:4.0.0
15721593
env:
15731594
ARGO_SERVER: ${{ needs.setup.outputs.argo-server }}
15741595
ARGO_HTTP1: ${{ needs.setup.outputs.argo-http1 }}
@@ -1625,21 +1646,21 @@ jobs:
16251646
id: create-test-arg
16261647
shell: bash
16271648
run: |
1628-
export comparing_variable="[]"
1629-
if [ "${{ inputs.marker }}" == "$comparing_variable" ]
1630-
then
1631-
TEST_ARG_M=""
1632-
else
1649+
TEST_ARG_M=""
1650+
EMPTY_MARKER="[]"
1651+
1652+
if [[ "${{ inputs.marker }}" != "$EMPTY_MARKER" ]]; then
16331653
TEST_ARG_M="-m"
16341654
fi
1655+
16351656
echo "test-arg=$TEST_ARG_M" >> "$GITHUB_OUTPUT"
16361657
- name: run-tests
16371658
id: run-tests
16381659
timeout-minutes: 340
16391660
continue-on-error: true
16401661
env:
16411662
ARGO_TOKEN: ${{ steps.get-argo-token.outputs.argo-token }}
1642-
uses: splunk/wfe-test-runner-action@v3.0
1663+
uses: splunk/wfe-test-runner-action@v5.0
16431664
with:
16441665
splunk: ${{ matrix.splunk.version }}${{ secrets.OTHER_TA_REQUIRED_CONFIGS }}
16451666
test-type: ${{ env.TEST_TYPE }}
@@ -1753,13 +1774,13 @@ jobs:
17531774
- uses: actions/upload-artifact@v4
17541775
if: ${{ !cancelled() }}
17551776
with:
1756-
name: archive splunk ${{ matrix.splunk.version }}${{ secrets.OTHER_TA_REQUIRED_CONFIGS }} ${{ env.TEST_TYPE }} ${{ matrix.vendor-version.image }} tests artifacts
1777+
name: archive splunk ${{ matrix.splunk.version }}${{ secrets.OTHER_TA_REQUIRED_CONFIGS }} ${{ env.TEST_TYPE }} ${{ matrix.vendor-version.image }} ${{ matrix.marker }} tests artifacts
17571778
path: |
17581779
${{ needs.setup.outputs.directory-path }}/test-results
17591780
- uses: actions/upload-artifact@v4
17601781
if: ${{ !cancelled() }}
17611782
with:
1762-
name: archive splunk ${{ matrix.splunk.version }}${{ secrets.OTHER_TA_REQUIRED_CONFIGS }} ${{ env.TEST_TYPE }} ${{ matrix.vendor-version.image }} tests logs
1783+
name: archive splunk ${{ matrix.splunk.version }}${{ secrets.OTHER_TA_REQUIRED_CONFIGS }} ${{ env.TEST_TYPE }} ${{ matrix.vendor-version.image }} ${{ matrix.marker }} tests logs
17631784
path: |
17641785
${{ needs.setup.outputs.directory-path }}/argo-logs
17651786
- name: Test Report
@@ -1797,7 +1818,7 @@ jobs:
17971818
splunk: ${{ fromJson(needs.meta.outputs.matrix_combinedSplunkversion) }}
17981819
os: [ "ubuntu:14.04", "ubuntu:16.04","ubuntu:18.04","ubuntu:22.04", "centos:7", "redhat:8.0", "redhat:8.2", "redhat:8.3", "redhat:8.4", "redhat:8.5" ]
17991820
container:
1800-
image: ghcr.io/splunk/workflow-engine-base:3.0.0
1821+
image: ghcr.io/splunk/workflow-engine-base:4.0.0
18011822
env:
18021823
ARGO_SERVER: ${{ needs.setup.outputs.argo-server }}
18031824
ARGO_HTTP1: ${{ needs.setup.outputs.argo-http1 }}
@@ -1868,7 +1889,7 @@ jobs:
18681889
continue-on-error: true
18691890
env:
18701891
ARGO_TOKEN: ${{ steps.get-argo-token.outputs.argo-token }}
1871-
uses: splunk/wfe-test-runner-action@v3.0
1892+
uses: splunk/wfe-test-runner-action@v5.0
18721893
with:
18731894
splunk: ${{ matrix.splunk.version }}${{ secrets.OTHER_TA_REQUIRED_CONFIGS }}
18741895
test-type: ${{ env.TEST_TYPE }}

‎README.md

Lines changed: 14 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -162,36 +162,31 @@ security-detect-secrets
162162

163163
**Description:**
164164

165-
- This action is intended as a Continuous Integration secret scan in an already "clean" repository. The default commit scan depth is the last 50 commits and can be adjusted using Custom Arguments
165+
- This action is intended as a Continuous Integration secret scan in an already "clean" repository.
166166

167-
- The stage checks for addition/deletion of any secret/sensitive data in last 50 commits of the repository.
167+
- The stage checks for addition/deletion of any secret/sensitive data in referenced commits (commits pushed or commits within PR).
168168

169-
**Action used** https://github.com/edplato/trufflehog-actions-scan
169+
**Action used** https://github.com/trufflesecurity/trufflehog
170170

171171
**Pass/fail behaviour**
172172

173-
- The stage is likely to fail if there is some sensitive or secrets or confidential data had been removed or added in the last 50 commits.
173+
- The stage is likely to fail if any sensitive secrets or confidential data were removed or added in the referenced commits.
174174

175175
**Troubleshooting steps for failures if any**
176176

177177
- User would need to update the commit history where the sensitive information is detected.
178178

179179
**Exception File**
180180

181-
- To ignore the file add the path of the file having the false positive in the `.github/workflows/exclude-patterns.txt`, ideally this should be avoided and only specific false positives should be added in exception files.
181+
- To ignore the file add the path of the file having the false positive in the `.github/workflows/exclude-patterns.txt`, ideally this should be avoided and only specific false positives should be added in exception files. This is file with newline separated regexes for files to exclude in scan.
182182

183183
- False positives include: public keys, random / dummy session keys or tokens.
184184

185-
- We can use this file `.github/workflows/trufflehog-false-positive.json` from action version `>=v0.9l-beta` to add specific failures or regexes.
186-
187-
- ref for how to add regex to json file : https://github.com/edplato/trufflehog-actions-scan#usage
188-
189-
- **NOTE:** The usage of `.github/workflows/trufflehog-false-positive.json` is not rolled out yet, PR for feature support: https://github.com/splunk/addonfactory-workflow-addon-release/pull/32
190-
185+
- User can add a `trufflehog:ignore` comment on the line containing the secret to ignore that secrets.
191186

192187
**Artifacts:**
193188

194-
- No additional artifacts, the commit info is available in the logs.
189+
- No additional artifacts, the commit info and secrets details are available in the logs.
195190

196191

197192
security-sast-semgrep
@@ -486,6 +481,9 @@ cim-compliance-report
486481
**Description**
487482

488483
- This stage does the setup for executing UI tests and reports the results
484+
- It is possible to parallelize UI tests execution by using pytest markers.
485+
To do so, one must specify `ui_marker` parameter in buid-test-release.yml as in [example](https://github.com/splunk/splunk-add-on-for-amazon-web-services/blob/925fd189737507dd91cc5275c59a8b390550411c/.github/workflows/build-test-release.yml#L35).
486+
Markers must be created prior and each test case must be marked (check [run-modinput-tests](#run-modinput-tests), and this [PR](https://github.com/splunk/splunk-add-on-for-amazon-web-services/pull/1237))
489487

490488
**Action used:**
491489
- No action used
@@ -525,6 +523,10 @@ Junit XML file
525523
**Description**
526524

527525
- This stage does the setup for executing Modinput tests and reports the results
526+
- It is possible to parallelize Modinput tests execution by using pytest markers.
527+
To do so, one must specify `marker` parameter in buid-test-release.yml as in [example](https://github.com/splunk/splunk-add-on-for-amazon-web-services/blob/603f37ee24565f23104c0297e55a0c72480f34c9/.github/workflows/build-test-release.yml#L33).
528+
Markers must be created prior and each test case must be marked (check the following references: [ref1](https://github.com/splunk/splunk-add-on-for-amazon-web-services/blob/main/tests/modinput_functional/README-test.md),
529+
[ref2](https://github.com/splunk/splunk-add-on-for-amazon-web-services/blob/main/tests/modinput_functional/centaurs/tacommon/test_ta_mark.py), [ref3](https://github.com/splunk/splunk-add-on-for-amazon-web-services/blob/603f37ee24565f23104c0297e55a0c72480f34c9/tests/modinput_functional/data_collection/aws_s3/test_aws_s3_line_content.py#L20))
528530

529531
**Action used:**
530532
- No action

‎renovate.json

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,5 @@
55
":semanticCommitTypeAll(chore)",
66
"schedule:earlyMondays",
77
":disableDependencyDashboard"
8-
],
9-
"ignoreDeps": [
10-
"edplato/trufflehog-actions-scan"
118
]
12-
}
9+
}

‎runbooks/publish_multiple_os_images_scripted_inputs.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Runbook to publish multiple images of different Linux flavors and versions for scripted inputs tests
2+
3+
Once there is new Splunk release, and [matrix](https://github.com/splunk/addonfactory-test-matrix-action) is updated, we need to make sure that Splunk images for scripted inputs tests are created and published.
4+
## Steps
5+
6+
### Update OS images
7+
- check what OS are listed in definition of matrix in scripted inputs tests [here](https://github.com/splunk/addonfactory-workflow-addon-release/blob/v4.16/.github/workflows/reusable-build-test-release.yml#L1966)
8+
- if any is missing in [ta-automation-docker-images](https://cd.splunkdev.com/taautomation/ta-automation-docker-images/-/tree/main/dockerfiles) then add new Dockerfile
9+
10+
### Create images and publish them to ECR
11+
- figure out what version of Splunk is needed (sha) using go/fetcher
12+
- trigger [pipeline](https://cd.splunkdev.com/taautomation/ta-automation-docker-images/-/pipelines/new) for every OS flavor separately

0 commit comments

Comments
 (0)