v5.3.4

v5.3.4

This is a patch update to bump a number of dependencies. No functional changes related to the tests themselves.

Improvements

• Remove logrus as direct dependency by @sebrandon1 in #2417
• Fix flaky test TestEvaluateAPICompliance by @sebrandon1 in #2430
• Brute force TestEvaluateAPICompliance unit test fix by @sebrandon1 in #2438

CI Updates

• Remove collector's sanity check from pre-main workflow by @shirmoran in #2420
• Add docker image cleanup to self-hosted YAMLs by @sebrandon1 in #2421
• Restrict OCP self-hosted PR runners by @sebrandon1 in #2437

Operator Scripting Updates

• Add upload results-spreadsheet certsuite sub-command by @shirmoran in #2389
• Make necessary results spreadsheet functions public by @shirmoran in #2442
• Add public constants for results spreadsheet by @shirmoran in #2447
• Add Setter and Getter functions to results spreadsheet google creds by @shirmoran in #2443

Dependency Updates

• Bump ubi9/ubi-minimal from 9.4-1227 to 9.4-1227.1725849298 by @dependabot in #2410
• Bump ubi9/ubi from 9.4-1214 to 9.4-1214.1725849297 by @dependabot in #2411
• Bump ubi9/python-39 from 1-197 to 1-197.1725907694 in /.github/actions/documentation by @dependabot in #2412
• Bump github.com/redhat-best-practices-for-k8s/oct from 0.0.21 to 0.0.22 by @dependabot in #2413
• Bump github.com/redhat-best-practices-for-k8s/certsuite-claim from 1.0.47 to 1.0.48 by @dependabot in #2414
• Bump github.com/redhat-best-practices-for-k8s/privileged-daemonset from 1.0.33 to 1.0.34 by @dependabot in #2415
• Update probe image to v0.0.8 by @sebrandon1 in #2416
• Bump golang.org/x/oauth2 from 0.22.0 to 0.23.0 by @dependabot in #2423
• Bump peter-evans/create-pull-request from 7.0.1 to 7.0.2 by @dependabot in #2422
• Bump k8s.io/apiextensions-apiserver from 0.31.0 to 0.31.1 by @dependabot in #2425
• Bump google.golang.org/api from 0.193.0 to 0.197.0 by @dependabot in #2426
• Bump helm.sh/helm/v3 from 3.15.4 to 3.16.1 by @dependabot in #2428
• Update yaml from v2 to v3 by @sebrandon1 in #2418
• Bump github.com/redhat-best-practices-for-k8s/privileged-daemonset from 1.0.34 to 1.0.35 by @dependabot in #2431
• Bump github.com/redhat-best-practices-for-k8s/oct from 0.0.22 to 0.0.23 by @dependabot in #2432
• Bump k8s.io/kubectl from 0.31.0 to 0.31.1 by @dependabot in #2433
• Bump github/codeql-action from 3.26.6 to 3.26.7 by @dependabot in #2436
• Bump github.com/k8snetworkplumbingwg/network-attachment-definition-client from 1.7.1 to 1.7.3 by @dependabot in #2434
• Bump peter-evans/create-pull-request from 7.0.2 to 7.0.3 by @dependabot in #2445

Full Changelog: v5.3.3...v5.3.4

v5.3.3

v5.3.3

In v5.3.3 we have added a few notable items:

• Fix for the operator-install-source test to correctly search through all available subscriptions for a valid subscription for cluster-wide operators.
• The observability suite has a new test that checks for future API version incompatibilities in workloads.

Improvements

• refactor : Renaming certsuite command tool by @bnshr in #2379
• refactor : Changes made for the change of config file name by @bnshr in #2400
• Fix some typos by @sebrandon1 in #2398
• Rename references from 'debug' to 'probe' by @sebrandon1 in #2397
• Add check to ensure workload compliance with the next k8s version by @tkrishtop in #2303
• Look through all subscriptions for install source by @sebrandon1 in #2407

Dependency Updates

• Bump ubi9/ubi from 9.4-1181 to 9.4-1181.1724035907 by @dependabot in #2382
• Bump github.com/Masterminds/semver/v3 from 3.2.1 to 3.3.0 by @dependabot in #2383
• Bump github/codeql-action from 3.26.5 to 3.26.6 by @dependabot in #2384
• Bump actions/upload-artifact from 4.3.6 to 4.4.0 by @dependabot in #2391
• Bump peter-evans/create-pull-request from 6.1.0 to 7.0.0 by @dependabot in #2392
• Bump ubi9/python-39 from 1-192.1724040313 to 1-197 in /.github/actions/documentation by @dependabot in #2396
• Bump ubi9/ubi from 9.4-1181.1724035907 to 9.4-1214 by @dependabot in #2393
• Bump ubi9/ubi-minimal from 9.4-1194 to 9.4-1227 by @dependabot in #2394
• Bump golang.org/x/term from 0.23.0 to 0.24.0 by @dependabot in #2395
• Bump github.com/redhat-best-practices-for-k8s/oct from 0.0.20 to 0.0.21 by @dependabot in #2404
• Bump peter-evans/create-pull-request from 7.0.0 to 7.0.1 by @dependabot in #2403
• Update Go to v1.23.1 by @sebrandon1 in #2402

CI Updates

• Update golangci settings by @sebrandon1 in #2386
• Avoid failing workflow when collectors sanity check fails by @shirmoran in #2390
• Temporarily disable collector sanity check by @sebrandon1 in #2399
• Remove bad ref to QE repo in ARM runs by @sebrandon1 in #2401
• Adjust docker to use /mnt sdb for storage in QE hosted by @sebrandon1 in #2406

New Contributors

• @tkrishtop made their first contribution in #2303

Full Changelog: v5.3.2...v5.3.3

v5.3.2

v5.3.2

A quick update to adjust compliant/non-compliant object logic in the operator test suite.

What's Changed

• Update probe to v0.0.7 by @sebrandon1 in #2378
• Fix OperatorAutomountTokens reversed result by @edcdavid in #2381

Full Changelog: v5.3.1...v5.3.2

v5.3.1

v5.3.1

This new version of the certsuite includes some more logging and logic improvements around gathering service accounts and specifically logging around the automount service account token tests in accesscontrol and operator suites.

Fixes and Improvements

• doc : Fix on topics by @bnshr in #2356
• Gather service accounts in claim file by @edcdavid in #2368

CI Updates

• Add ARM64 based QE runner by @sebrandon1 in #2363

Dependency Updates

• Update Go to v1.22.6 by @sebrandon1 in #2360
• Update openshift libraries manually by @sebrandon1 in #2361
• Update GolangCI-lint to v1.60.1 by @sebrandon1 in #2362
• Bump ubi9/python-39 from 1-192.1723128185 to 1-192.1724040313 in /.github/actions/documentation by @dependabot in #2365
• Bump github/codeql-action from 3.26.3 to 3.26.4 by @dependabot in #2366
• Update GolangCI-lint to v1.60.2; various fixes by @sebrandon1 in #2367
• Update GolangCI-lint to v1.60.3 by @sebrandon1 in #2372
• Bump github.com/redhat-best-practices-for-k8s/oct from 0.0.19 to 0.0.20 by @dependabot in #2375
• Bump github.com/redhat-best-practices-for-k8s/certsuite-claim from 1.0.45 to 1.0.46 by @dependabot in #2376
• Bump github.com/redhat-best-practices-for-k8s/privileged-daemonset from 1.0.31 to 1.0.33 by @dependabot in #2377
• Bump github/codeql-action from 3.26.4 to 3.26.5 by @dependabot in #2374
• Bump github.com/mittwald/go-helm-client from 0.12.12 to 0.12.13 by @dependabot in #2373

Full Changelog: v5.3.0...v5.3.1

v5.3.0

v5.3.0

We have completed an organization rename:

• test-network-function --> redhat-best-practices-for-k8s

This has affected changes across all of our repositories to change all of our go modules, etc. We have also changed all of our image repositories on quay as well:

https://quay.io/organization/redhat-best-practices-for-k8s

Improvements

• Add option to sanitize claim file result output by @sebrandon1 in #2299
• Remove unused func in claimhelper by @sebrandon1 in #2306
• Revert "Remove unused func in claimhelper" by @sebrandon1 in #2309
• Add claimhelper pkg unit tests by @sebrandon1 in #2307
• Various CI Updates by @sebrandon1 in #2308
• Enable observability suite for PR OCP testing by @sebrandon1 in #2311
• Rename project to 'certsuite' by @sebrandon1 in #2321
• Fix unstable push; add legacy image to tnf image yaml by @sebrandon1 in #2324
• Fix DEBUG_ variables by @sebrandon1 in #2326
• Fix for the false positive of access-control's capabilities tcs. by @greyerof in #2352
• Fix badges after repo rename by @rdavid in #2336
• GetOwnerReferences does not return error by @rdavid in #2333
• Updated URL for github docs. by @greyerof in #2334
• cli: update Certsuite banner to v5.3 by @jmontesi in #2335

Dependency Updates

• Update operator-sdk to v1.36.0 by @sebrandon1 in #2304
• Update operator-sdk to v1.36.1 by @sebrandon1 in #2330
• Bump github.com/test-network-function/oct from 0.0.16 to 0.0.17 by @dependabot in #2305
• Bump ubi9/python-39 from 1-192 to 1-192.1722518946 in /.github/actions/documentation by @dependabot in #2314
• Bump actions/upload-artifact from 4.3.4 to 4.3.5 by @dependabot in #2312
• Bump github.com/k8snetworkplumbingwg/network-attachment-definition-client from 1.7.0 to 1.7.1 by @dependabot in #2313
• Bump golang.org/x/term from 0.22.0 to 0.23.0 by @dependabot in #2317
• Bump github.com/mittwald/go-helm-client from 0.12.10 to 0.12.11 by @dependabot in #2316
• Bump github/codeql-action from 3.25.15 to 3.26.0 by @dependabot in #2318
• Bump actions/upload-artifact from 4.3.5 to 4.3.6 by @dependabot in #2319
• Bump actions/upload-artifact from 4.3.5 to 4.3.6 by @dependabot in #2327
• Bump github.com/mittwald/go-helm-client from 0.12.11 to 0.12.12 by @dependabot in #2328
• Bump github.com/docker/docker from 26.1.4+incompatible to 26.1.5+incompatible by @dependabot in #2329
• Update debug image to v0.0.4 by @sebrandon1 in #2315
• Bump docker/build-push-action from 6.5.0 to 6.6.1 by @dependabot in #2323
• Bump github/codeql-action from 3.26.2 to 3.26.3 by @dependabot in #2358
• Bump ubi9/python-39 from 1-192.1722518946 to 1-192.1723128185 in /.github/actions/documentation by @dependabot in #2359
• Bump github.com/redhat-best-practices-for-k8s/oct from 0.0.18 to 0.0.19 by @dependabot in #2349
• Bump github.com/operator-framework/api from 0.26.0 to 0.27.0 by @dependabot in #2353
• Bump docker/build-push-action from 6.6.1 to 6.7.0 by @dependabot in #2343
• Bump helm.sh/helm/v3 from 3.15.3 to 3.15.4 by @dependabot in #2345
• Bump github/codeql-action from 3.26.0 to 3.26.2 by @dependabot in #2346
• Update k8s deps + controller runtime by @sebrandon1 in #2350
• Update certsuite-probe to v0.0.6 by @sebrandon1 in #2337
• Update operator-sdk to v1.36.1 by @sebrandon1 in #2330

Full Changelog: v5.2.3...v5.3.0

v5.2.3

v5.2.3

The v5.2.3 release of the test suite has a couple of notable changes such as a fix for the PDB test case and marking the read-only filesystem operator test as an 'Optional' test case.

This is also the first release that is using the new k8s-best-practices-debug image for the debug pod that spawns as part of the test suite.

Test Case Changes

• Mark read-only filesystem test as Optional by @sebrandon1 in #2245
• tests/observability: fix observability-pod-disruption-budged test case by @jmontesi in #2285

Improvements

• Add scaling helper unit tests by @sebrandon1 in #2249
• cmd/certsuite: update default value for log file in "check results" by @jmontesi in #2251
• Add 'new' image tag to push during CI runs by @sebrandon1 in #2254
• cmd/certsuite: refactor main to allow unit testing subcommands by @jmontesi in #2259
• cmd/certsuite: move the "list" flag from "run" to "info" by @jmontesi in #2264
• internal/datautil: add new package for basic data handling by @jmontesi in #2279
• Remove RELEASE_LEVEL by @sebrandon1 in #2283
• Change debug-partner to k8s-best-practices-debug by @sebrandon1 in #2282
• use ./certsuite instead of ./tnf (no longer exists) by @edcdavid in #2289
• Remove result cast; -claim update by @sebrandon1 in #2296

Documentation Updates

• refactor : Update cnf doc link to k8s doc link by @bnshr in #2257
• Fix broken link for tnf_config.yml by @vikasmulaje in #2280
• Add data collection doc by @shirmoran in #2281

Dependency Updates

• Bump github.com/test-network-function/oct from 0.0.14 to 0.0.15 by @dependabot in #2246
• Bump github/codeql-action from 3.25.11 to 3.25.12 by @dependabot in #2248
• Bump github.com/test-network-function/privileged-daemonset from 1.0.27 to 1.0.28 by @dependabot in #2247
• Bump docker/build-push-action from 6.3.0 to 6.4.0 by @dependabot in #2255
• Bump github.com/test-network-function/test-network-function-claim from 1.0.41 to 1.0.42 by @dependabot in #2256
• Update preflight to v1.10.0 by @sebrandon1 in #2260
• Missed version update for golangci by @sebrandon1 in #2261
• Bump docker/build-push-action from 6.4.0 to 6.4.1 by @dependabot in #2262
• Bump k8s.io/client-go from 0.30.2 to 0.30.3 by @dependabot in #2267
• Bump k8s.io/apiextensions-apiserver from 0.30.2 to 0.30.3 by @dependabot in #2268
• Bump k8s.io/kubectl from 0.30.2 to 0.30.3 by @dependabot in #2266
• Bump github/codeql-action from 3.25.12 to 3.25.13 by @dependabot in #2270
• Bump docker/setup-buildx-action from 3.4.0 to 3.5.0 by @dependabot in #2271
• Bump docker/login-action from 3.2.0 to 3.3.0 by @dependabot in #2273
• Bump docker/setup-qemu-action from 3.1.0 to 3.2.0 by @dependabot in #2272
• Bump docker/build-push-action from 6.4.1 to 6.5.0 by @dependabot in #2274
• Bump github.com/test-network-function/oct from 0.0.15 to 0.0.16 by @dependabot in #2275
• Bump ubi9/ubi from 9.4-1123.1719560047 to 9.4-1181 by @dependabot in #2288
• Bump ubi9/ubi-minimal from 9.4-1134 to 9.4-1194 by @dependabot in #2287
• Bump ubi9/python-39 from 1-186.1720018722 to 1-192 in /.github/actions/documentation by @dependabot in #2286
• Bump docker/setup-buildx-action from 3.5.0 to 3.6.1 by @dependabot in #2297
• Bump github.com/test-network-function/privileged-daemonset from 1.0.28 to 1.0.29 by @dependabot in #2298
• Bump github.com/docker/docker from 25.0.5+incompatible to 26.1.4+incompatible by @dependabot in #2300
• Bump golangci/golangci-lint-action from 6.0.1 to 6.1.0 by @dependabot in #2301
• Bump github/codeql-action from 3.25.14 to 3.25.15 by @dependabot in #2293
• Bump ossf/scorecard-action from 2.3.3 to 2.4.0 by @dependabot in #2292
• Bump github.com/test-network-function/test-network-function-claim from 1.0.42 to 1.0.43 by @dependabot in #2294
• Bump github/codeql-action from 3.25.13 to 3.25.14 by @dependabot in #2290

New Contributors

• @vikasmulaje made their first contribution in #2280

Full Changelog: v5.2.2...v5.2.3

v5.2.2

v5.2.2

This was a quick turnaround release where we did some notable changes:

• Changed the access-control-namespace test from Mandatory to Optional for all scenarios.
• Released a new version of the parser with v0.4.7. Found here.

Note: There were no changes in the -partner repository, so the partner version.json did not need an update at this point.

Improvements

• Bump parser version to v0.4.7 by @edcdavid in #2243
• Adjust 'access-control-namespace' test to be Optional by @sebrandon1 in #2242
• Bump parser version to v0.4.6 by @edcdavid in #2238
• Repository folder structure reorganization by @jmontesi in #2237

Dependency Updates

• Bump actions/setup-go from 5.0.1 to 5.0.2 by @dependabot in #2239
• Bump helm.sh/helm/v3 from 3.15.2 to 3.15.3 by @dependabot in #2241

Full Changelog: v5.2.1...v5.2.2

v5.2.1

v5.2.1

This version of the certsuite introduces 4 new operator suite tests:

• testOperatorPodsRunAsUserID(): This test verifies that no pods managed by operators run with the root user ID (UID) of 0, which could introduce security vulnerabilities.
• testOperatorPodsRunAsNonRoot(): This test ensures that pods managed by operators adhere to security best practices by running as non-root users.
• testOperatorPodsAutomountTokens(): This test evaluates the configuration of automount service tokens in pods managed by operators.
• testOperatorContainersReadOnlyFilesystem(): This test verifies whether containers within pods managed by operators have a read-only root filesystem, enhancing security by preventing unauthorized modifications.

New Operator Tests

• security requirements of the container-native operators by @shimritproj in #1967

Improvements

• Add check pkg unit tests by @sebrandon1 in #2196
• fix tnf_config.yaml directory typo batch script command by @shirmoran in #2214
• configure : Test operator labels in tnf_config yaml by @bnshr in #2213
• cmd/certsuite: add new command to show the version by @jmontesi in #2219
• Add deployment pkg unit tests by @sebrandon1 in #2215
• cmd/certsuite: add new "certsuite info" command to display Catalog info by @jmontesi in #2228

Dependency Updates

• Bump github/codeql-action from 3.25.10 to 3.25.11 by @dependabot in #2211
• Bump ubi9/python-39 from 1-186 to 1-186.1719562233 in /.github/actions/documentation by @dependabot in #2217
• Bump ubi9/ubi from 9.4-1123 to 9.4-1123.1719560047 by @dependabot in #2218
• Bump docker/setup-qemu-action from 3.0.0 to 3.1.0 by @dependabot in #2221
• Bump docker/build-push-action from 6.2.0 to 6.3.0 by @dependabot in #2220
• Bump docker/setup-buildx-action from 3.3.0 to 3.4.0 by @dependabot in #2222
• Bump actions/download-artifact from 4.1.7 to 4.1.8 by @dependabot in #2225
• Bump actions/upload-artifact from 4.3.3 to 4.3.4 by @dependabot in #2226
• Bump golang.org/x/term from 0.21.0 to 0.22.0 by @dependabot in #2224
• Bump certifi from 2024.2.2 to 2024.7.4 in /.github/actions/documentation by @dependabot in #2227
• Update Go to v1.22.5 by @sebrandon1 in #2229
• Bump github.com/test-network-function/oct from 0.0.12 to 0.0.14 by @dependabot in #2232
• Bump zipp from 3.18.1 to 3.19.1 in /.github/actions/documentation by @dependabot in #2234
• Bump github.com/test-network-function/test-network-function-claim from 1.0.39 to 1.0.41 by @dependabot in #2233
• Bump ubi9/python-39 from 1-186.1719562233 to 1-186.1720018722 in /.github/actions/documentation by @dependabot in #2230
• Update GolangCI-lint to v1.59.1 by @sebrandon1 in #2216

Full Changelog: v5.2.0...v5.2.1

v5.2.0

v5.2.0

With the v5.2.0 release of the test suite, we have done some underlying changes to the way the test suite is ran. Mainly, we have removed the bash scripts that were previously needed to kick things off. Now all you need to run the test suite is our new certsuite binary. We will also be publishing the binaries on release built for different architectures.

The documentation and the README have been updated with this new information as well.

Script Removal

• docs: copy friendly version of the docker cmd to run the test suite by @jmontesi in #2204
• Delete legacy code and avoid using environment variables by @jmontesi in #2207
• Update the Certsuite demo by @jmontesi in #2199
• cli: update Certsuite banner to v5.2 by @jmontesi in #2193

CI Updates

• workflows: add new workflow to upload release assets by @jmontesi in #2208
• workflows: several fixes to upload release assets by @jmontesi in #2209

Dependency Updates

• Bump github.com/test-network-function/privileged-daemonset from 1.0.26 to 1.0.27 by @dependabot in #2206

Full Changelog: v5.1.3...v5.2.0

v5.1.3

v5.1.3

Brings in a fix for the IsRedHatRelease test to allow for images that somehow show they are "Beta".

Improvements

• feat : Add helper function to get all operator pods by @bnshr in #2130
• Fix variable name by @bnshr in #2180
• Dockerfile: do not build the legacy binary by @jmontesi in #2181
• Delete legacy bash scripts to run the Best Practices Test Suite by @jmontesi in #2186
• Fix batch script to enable run on an entire catalog by @shirmoran in #2167
• Add logging; unit tests for UBI9 by @sebrandon1 in #2191
• Adjust RHEL regex to support Beta versions by @sebrandon1 in #2194
• Adjust typos by @sebrandon1 in #2195
• Remove unnecessary kubeconfig check by @shirmoran in #2198

Documentation Updates

• docs: update to show how to run the test suite with the "certsuite" cmd by @jmontesi in #2174
• docs: move and rename docs on how to run the Test Suite inside a cluster pod by @jmontesi in #2192

CI Updates

• Makefile: use curl to get the parser webpage (results.html) by @jmontesi in #2175
• Add 4.17 nightly YAML by @sebrandon1 in #2187
• Add 4.17 to RHCOS script by @sebrandon1 in #2189

Dependency Updates

• Update operator-sdk to v1.35.0 by @sebrandon1 in #2184
• Bump github.com/mittwald/go-helm-client from 0.12.9 to 0.12.10 by @dependabot in #2200

Github Actions Updates

• Bump peter-evans/create-pull-request from 6.0.5 to 6.1.0 by @dependabot in #2177
• Bump docker/build-push-action from 6.0.0 to 6.0.1 by @dependabot in #2178
• Bump docker/build-push-action from 6.0.1 to 6.0.2 by @dependabot in #2182
• Bump ludeeus/action-shellcheck from cd81f4475ab741e097ec0fe73b692f3e49d66b8c to 00b27aa7cb85167568cb48a3838b75f4265f2bca by @dependabot in #2183
• Bump docker/build-push-action from 6.0.2 to 6.1.0 by @dependabot in #2188
• Bump docker/build-push-action from 6.1.0 to 6.2.0 by @dependabot in #2201

Full Changelog: v5.1.2...v5.1.3