v5.0.1

v5.0.1 is a patch-level update to bring improved resource allocation for the daemonset the test suite spawns as part of testing.

There are also a number of dependency updates.

Daemonset Resource Usage Improvements

• Adding configurable cpu requests/limit for the debug daemonset by @edcdavid in #1805

Logging Improvements

• log: review logs for autodiscover and provider packages by @jmontesi in #1811

CI Updates

• Add retries for 'make build-image-local' by @sebrandon1 in #1812
• Re-enable gradetool by @sebrandon1 in #1810
• Remove free disk space cleanup by @sebrandon1 in #1821

Dependency Updates

• Update preflight to v1.8.0 by @sebrandon1 in #1809
• build(deps): bump helm.sh/helm/v3 from 3.13.3 to 3.14.0 by @dependabot in #1816
• build(deps): bump k8s.io/kubectl from 0.29.0 to 0.29.1 by @dependabot in #1813
• build(deps): bump ubi8/ubi from 8.9-1107 to 8.9-1107.1705420509 by @dependabot in #1818
• build(deps): bump k8s.io/apiextensions-apiserver from 0.29.0 to 0.29.1 by @dependabot in #1817
• build(deps): bump ubi8/ubi-minimal from 8.9-1108 to 8.9-1108.1705420507 by @dependabot in #1819
• build(deps): bump github.com/test-network-function/privileged-daemonset from 1.0.19 to 1.0.20 by @dependabot in #1822
• build(deps): bump github.com/k8snetworkplumbingwg/network-attachment-definition-client from 1.4.0 to 1.5.0 by @dependabot in #1823
• Update images to ubi9/ubi9-minimal by @sebrandon1 in #1825

Full Changelog: v5.0.0...v5.0.1

v5.0.0

v5.0.0 - No More Ginkgo

Our first release of 2024 is a big one. We are releasing our first v5.x.x release, now without the ginkgo dependency. A lot of work has gone into removing this underlying dependency from the test suite. As far as the end-user knows, everything should act very similar to the previous v4.x.x releases.

The CLI has undergone a good quality-of-life update as well which now shows each test and its corresponding Passed/Failed/Skipped state in realtime.

Bug Fixes

• Fix for the webserver kubeconfig file handling. by @greyerof in #1625
• Graceful exit when no kubeconfig is configured by @edcdavid in #1668
• Revert "Only capture ctrl-c while tests are running" by @edcdavid in #1693
• Only capture ctrl-c while tests are running by @edcdavid in #1694
• Only capture ctrl-c while tests are running by @edcdavid in #1699
• Move flags processing to run function by @edcdavid in #1696
• The find command requires a directory as the first argument by @rdavid in #1709
• Add operator versions by @rdavid in #1708
• Rename poststart test by @edcdavid in #1698
• Fix for fatal error in diagnostics by @edcdavid in #1715
• give precedence to the KUBECONFIG variable instead of the $HOME/.kube/config kubeconfig by @edcdavid in #1755
• autodiscover: delete deprecated default operator label by @jmontesi in #1760
• Bug fix: adjust logic in testPodClusterRoleBindings by @jmontesi in #1792

WebServer Improvements

• webserver: add implementation for uploading tnf config file and add all the fields by @aabughosh in #1644
• webserver: add tests as a table allowing the user to select an option by @aabughosh in #1662

Unit Tests

• Remove gotools; adjust expected/actual by @sebrandon1 in #1667
• Additional unit test coverage by @sebrandon1 in #1780

Documentation Improvements

• docs: add CNFCERT demo by @jmontesi in #1757
• docs: fix CNFCERT demo path by @jmontesi in #1758

Claim File Output Improvements

• Switch to 1-to-1 TestID to Result struct by @sebrandon1 in #1666
• re-add logs in claim file by @edcdavid in #1725
• flags: change "claimloc" for "output-dir" by @jmontesi in #1726
• version: bump claim file format version by @jmontesi in #1738

CLI Improvements

• cli: display information about the CNFCERT run by @jmontesi in #1690
• CLI output improvements: check's elapsed time and last log trace. by @greyerof in #1745
• CLI output: non-tty mode improvements. by @greyerof in #1750
• cli: fix typo in skip reason by @jmontesi in #1756
• cli: cover all possible skip paths to print it in the CLI output by @jmontesi in #1765

Logging Improvements

• log: add new log package that uses slog as global logger by @jmontesi in #1663
• log: remove logrus and migrate all logs to slog by @jmontesi in #1703
• log: add log file to the results tar archive by @jmontesi in #1710
• log: set the configured log level to multiloggers by @jmontesi in #1711
• log: remove loghelper package by @jmontesi in #1721
• observability: review logs by @jmontesi in #1761
• manageability: review logs by @jmontesi in #1770
• operator: review logs by @jmontesi in #1772
• performance: review logs by @jmontesi in #1777
• networking: review logs by @jmontesi in #1782
• certification: review logs by @jmontesi in #1783
• access-control: review logs by @jmontesi in #1787
• platform: review logs by @jmontesi in #1791
• lifecycle: logs review by @jmontesi in #1789
• preflight: review logs by @jmontesi in #1793

Parser Improvements

• Bump parser version to v0.2.0 by @edcdavid in #1676
• extracts results.html from tar.gz and debug mode by @edcdavid in #1706
• update the version of the claim release by @aabughosh in #1784

CI Improvements

• Switch gradetool image to latest by @sebrandon1 in #1681
• Enable QE nightlies for ginkgo_removal by @sebrandon1 in #1723
• Revert "Enable QE nightlies for ginkgo_removal" by @sebrandon1 in #1724
• Enable QE PR tests by @sebrandon1 in #1748
• Remove self-hosted, kind-based QE nightlies by @sebrandon1 in #1776
• Add retries for -partner setup calls by @sebrandon1 in #1775
• Adjust the directories cleaned up in QE nightlies by @sebrandon1 in #1806

Operator Tool Improvements

• Updating Telco list by @edcdavid in #1712
• skip operator if claim file couldn't be parsed by @edcdavid in #1714
• batch operator script - fix long term errors by @edcdavid in #1741

ginkgo_removal work

• Dependency updates for ginkgo removal branch by @sebrandon1 in #1615
• ginkgo_removal branch: enable github checks on new PRs. by @greyerof in #1608
• Fixed deadlock in check.SetResult(). by @greyerof in #1619
• Transition manageability suite to ginkgoless by @sebrandon1 in #1614
• Transition performance suite to ginkgoless. by @greyerof in #1620
• Transition certification suite to ginkgoless. by @greyerof in #1622
• Rebase the ginkgo_removal branch with main by @sebrandon1 in #1642
• Allow more than one skip functions per check. by @greyerof in #1645
• Transition accesscontrol suite to ginkgoless by @sebrandon1 in #1616
• Transition networking suite to ginkgoless by @sebrandon1 in #1621
• Transition lifecycle suite to ginkgoless by @sebrandon1 in #1627
• Lifecycle ts: fix to avoid some checks to be skipped. by @greyerof in #1655
• Load the checks' database using a standard function instead of...

v4.5.7

v4.5.7 is a patch release of the CNF Test Suite.

We improved the test surrounding cluster-role-bindings in #1646 for pods that are owned by a cluster wide operator.

We also bumped the claimVersion to v0.2.0 as there is a new JSON structure update to the claim JSON.

Test Improvements

• Allow cluster role bindings for pods ultimately owned by a cluster wide operator by @edcdavid in #1646
• Fix WaitGroup Done() call by @bnshr in #1688

Claim File Updates

• Cherry pick 1-to-1 result struct change by @sebrandon1 in #1675

Dependency Updates

• Bump github.com/mittwald/go-helm-client from 0.12.3 to 0.12.4 by @dependabot in #1652
• Bump github.com/deckarep/golang-set/v2 from 2.3.1 to 2.4.0 by @dependabot in #1656
• Bump github.com/operator-framework/api from 0.19.0 to 0.20.0 by @dependabot in #1657
• Bump github.com/onsi/ginkgo/v2 from 2.13.1 to 2.13.2 by @dependabot in #1671
• Bump github.com/deckarep/golang-set/v2 from 2.4.0 to 2.5.0 by @dependabot in #1674

CI Improvements

• Temp. disable collector sanity check by @sebrandon1 in #1653
• Add depends-on PRs action to qe-hosted workflow by @sebrandon1 in #1677
• Add-batch-cert-script by @edcdavid in #1604
• Switch gradetool to latest image (main) by @sebrandon1 in #1680
• Increase QE timeout to 90 minutes by @sebrandon1 in #1683
• Fix copyright years by @rdavid in #1686
• Remove GPL commitment by @sebrandon1 in #1689

Full Changelog: v4.5.6...v4.5.7

v4.5.6

Bug Fixes

• Fix catalog links by @edcdavid in #1650
• Fixed function AreCPUResourcesWholeUnits(). by @greyerof in #1631

Dependency Updates

• Update Go to v1.21.4 by @sebrandon1 in #1598
• Bump github.com/gorilla/websocket from 1.4.2 to 1.5.1 by @dependabot in #1596
• Bump github.com/onsi/ginkgo/v2 from 2.13.0 to 2.13.1 by @dependabot in #1610
• Bump github.com/test-network-function/privileged-daemonset from 1.0.14 to 1.0.15 by @dependabot in #1612
• Bump github.com/test-network-function/test-network-function-claim from 1.0.30 to 1.0.31 by @dependabot in #1611
• Bump ubi8/ubi from 8.8-1067.1698056881 to 8.9-1028 by @dependabot in #1633
• Bump github.com/test-network-function/oct from 0.0.3 to 0.0.4 by @jmontesi in #1632
• Bump ubi8/ubi-minimal from 8.8-1072.1697626218 to 8.9-1029 by @dependabot in #1634
• Bump k8s.io/kubectl from 0.28.3 to 0.28.4 by @dependabot in #1636

CI Improvements

• Modify ENV vars for config and report dirs by @sebrandon1 in #1602
• Disable Go cache for self-hosted by @sebrandon1 in #1626
• Fix gchat alert message. by @greyerof in #1630
• Re-enable QE PR checks by @sebrandon1 in #1640
• enable collector's sanity check by @shirmoran in #1606

Full Changelog: v4.5.5...v4.5.6

v4.5.5

Bug Fixes

• Do not consider the exec probes in performance-rt-apps-no-exec-probes by @edcdavid in #1590
• Fix regression in performance-rt-apps-no-exec-probes by @edcdavid in #1593

New Website

• website for our cnf by @aabughosh in #1544

Parser Updates

• Getting bugfix for parser (still needing bootstrap for some elements) by @edcdavid in #1595

Dependency Updates

• Bump github.com/spf13/cobra from 1.7.0 to 1.8.0 by @dependabot in #1588
• Bump github.com/fatih/color from 1.15.0 to 1.16.0 by @dependabot in #1589
• Updating the HTML parser to the latest version (v0.1.3) by @edcdavid in #1587
• Bump helm.sh/helm/v3 from 3.13.1 to 3.13.2 by @dependabot in #1592

CI Updates

• Setup QE OCP 4.14 runner by @sebrandon1 in #1594

Full Changelog: v4.5.4...v4.5.5

v4.5.4

Small update to include a fix for the pod-role-bindings test.

Bug Fixes

• Allow Role bindings across namespaces by @edcdavid in #1583

Dependency Updates

• Bump github.com/operator-framework/api from 0.18.0 to 0.19.0 by @dependabot in #1584

Full Changelog: v4.5.3...v4.5.4

v4.5.3

v4.5.3 is a smaller update with some vulnerability dependency updates and a bug fix to the performance suite.

Bug Fixes

• performance: avoid failing test case if a process cannot be found by @jmontesi in #1564

Improvements

• Add OCP 4.14 to the compatibility matrix by @sebrandon1 in #1574

Dependency Updates

• Bump github.com/go-logr/logr from 1.2.4 to 1.3.0 by @dependabot in #1565
• Bump github.com/docker/docker from 24.0.6+incompatible to 24.0.7+incompatible by @dependabot in #1568
• Bump go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp from 0.41.1 to 0.44.0 by @dependabot in #1530
• Bump go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp from 0.42.0 to 0.44.0 by @dependabot in #1572
• Bump google.golang.org/grpc from 1.58.2 to 1.58.3 by @dependabot in #1571
• Bump github.com/test-network-function/privileged-daemonset from 1.0.13 to 1.0.14 by @dependabot in #1579
• Bump github.com/test-network-function/test-network-function-claim from 1.0.29 to 1.0.30 by @dependabot in #1580
• Bump github.com/test-network-function/oct from 0.0.2 to 0.0.3 by @dependabot in #1581

CI Improvements

• Add cron to run preflight against unstable image by @sebrandon1 in #1569
• Add YAML for intrusive QE OCP testing by @sebrandon1 in #1573
• enable run of collector's sanity check in CI by @shirmoran in #1567
• Update GolangCI-lint to v1.55.1 by @sebrandon1 in #1575
• Update _typos.toml by @sebrandon1 in #1577

Full Changelog: v4.5.2...v4.5.3

v4.5.2

v4.5.2 is a smaller update with a bunch of dependency updates and CI improvements. Nothing functional as far as test cases are concerned.

Dependency Updates

• Update preflight to v1.7.1 by @sebrandon1 in #1545
• Bump ubi8/ubi-minimal from 8.8-1072.1696517598 to 8.8-1072.1697626218 by @dependabot in #1548
• Bump ubi8/ubi from 8.8-1067.1696517599 to 8.8-1067.1697633337 by @dependabot in #1547
• Bump k8s.io/kubectl from 0.28.2 to 0.28.3 by @dependabot in #1546
• Bump ubi8/ubi from 8.8-1067.1697633337 to 8.8-1067.1698056881 by @dependabot in #1554
• Bump github.com/operator-framework/api from 0.17.7 to 0.18.0 by @dependabot in #1553
• Bump google.golang.org/grpc from 1.55.0 to 1.56.3 by @dependabot in #1559
• Update Go to v1.21.3 by @sebrandon1 in #1558
• Bump github.com/test-network-function/test-network-function-claim from 1.0.28 to 1.0.29 by @dependabot in #1560
• Bump github.com/test-network-function/oct from 0.0.1 to 0.0.2 by @dependabot in #1561
• Prepare for v4.5.2 by @sebrandon1 in #1563
• Bump github.com/test-network-function/privileged-daemonset from 1.0.12 to 1.0.13 by @dependabot in #1562

CNF Suite Operator Work

• Allow CNF Cert Suite app's go-clients to work in a k8s pod. by @greyerof in #1543
• Added yaml template to deploy the CNF Cert Suite in a kubernetes/Openshift cluster. by @greyerof in #1557

CI Improvements

• linters: add some missing linters to the Makefile by @jmontesi in #1552
• linters: use golangci-lint GitHub action instead of a manual setup by @jmontesi in #1556
• Add OCP QE Runner by @sebrandon1 in #1549

QE Coverage Update

• QE Coverage : Mark dpdk cpu pinning exec probe QE implemented by @bnshr in #1466

Full Changelog: v4.5.1...v4.5.2

v4.5.1

New to v4.5.1 is the TNF Config Generator introduced in #1506. This allows users to build a complete TNF config file without having to touch YAML.

There has been a bunch of documentation written about this tool as well so be sure to check that out here.

TNF Config Updates

• config: add info and a quick demo of the CNF Config Generator by @jmontesi in #1520
• config: allow interative svg image for the CNF Config Generator demo by @jmontesi in #1526
• config: fix path for the CNF Config Generator demo image by @jmontesi in #1527
• cmd/tnf: new tool to generate the CNF config file with user input by @jmontesi in #1506
• config: delete unused struct definitions by @jmontesi in #1514
• config: add two more options to the CNF config generator by @jmontesi in #1523
• config: update documentation by @jmontesi in #1525

Test Improvements

• check for local storage provisioners for SNO clusters by @edcdavid in #1507
• Making the lifecycle-pod-scheduling test optional in telco scenarios by @edcdavid in #1521

Claim Compare Tool

• Official documentation for the claim compare tool. by @greyerof in #1533

Dependency Updates

• Update -preflight to v1.7.0 by @sebrandon1 in #1496
• Update operator-sdk to v1.32.0 by @sebrandon1 in #1505
• Update Go to v1.21.2 by @sebrandon1 in #1504
• Bump ubi8/ubi from 8.8-1067 to 8.8-1067.1696517599 by @dependabot in #1512
• Bump ubi8/ubi-minimal from 8.8-1072 to 8.8-1072.1696517598 by @dependabot in #1511
• Bump golang.org/x/net from 0.14.0 to 0.17.0 by @dependabot in #1531
• Bump github.com/test-network-function/privileged-daemonset from 1.0.10 to 1.0.12 by @dependabot in #1542
• Bump helm.sh/helm/v3 from 3.13.0 to 3.13.1 by @dependabot in #1538
• Bump github.com/test-network-function/test-network-function-claim from 1.0.27 to 1.0.28 by @dependabot in #1537
• Bump github.com/onsi/ginkgo/v2 from 2.12.1 to 2.13.0 by @dependabot in #1536
• Bump k8s.io/apiextensions-apiserver from 0.28.2 to 0.28.3 by @dependabot in #1539

CI Improvements

• Send gchat alert message to dev team if unstable image creation failed. by @greyerof in #1500
• Fix for rhcos_versions.sh script. by @greyerof in #1502
• Send gchat alert message to dev team if release image creation failed. by @greyerof in #1503
• depends-on-action: no need for changes in Makefile anymore by @fredericlepied in #1509
• Removed local certdb and use it from OCT. by @greyerof in #1513
• Disable DEBUG_TNF flag in QE runs by @sebrandon1 in #1510
• Enable flaky retries for QE by @sebrandon1 in #1528
• Added to CI tests sanity checks for the collector by @shirmoran in #1365

Full Changelog: v4.5.0...v4.5.1

v4.5.0

v4.5.0

There are non-backwards compatible changes as part of this release, hence the reason for moving from v4.4.x to v4.5.x. See the section below about the TNF Config Changes.

In addition to the config changes there has been functionality removed surrounding the container certification status test that was a piece of older functionality that was rarely used. The way it used to check images outside of the labeling scheme, relying on the TNF config to house the image names is not the way we wanted users to interact with the test suite.

TNF Config Changes

• configuration: delete the init function by @jmontesi in #1443
• config: delete deprecated field targetPodLabels by @jmontesi in #1390
• config: delete static container image check by @jmontesi in #1455
• config: omit empty options in the TNF config yaml file by @jmontesi in #1486

New Additions

• New tool to verify the certification status of container images by @jmontesi in #1450
• Comparison tool: compare node's labels and annotations. by @greyerof in #1444
• Claim comparison tool: compare versions. by @greyerof in #1480

Fixes and Improvements

Report Objects

• Fix report objects for toleration bypass test by @sebrandon1 in #1437
• Add 'reason' in report object for nodeport test by @sebrandon1 in #1436
• Fix report objects for sysctl config test by @sebrandon1 in #1433
• Fix spacing in exec probe output by @sebrandon1 in #1438
• Add the name of the object responsible for skipping in addition to type by @edcdavid in #1440
• Fix report objects for namespace prefix test by @sebrandon1 in #1434
• Fix report objects for hostpath test by @sebrandon1 in #1435

Certified Container Changes

• Fix: run container-is-certified also in the case of digest only images by @edcdavid in #1439
• Remove skips for testContainerCertificationStatus by @sebrandon1 in #1462
• Fix container status by @edcdavid in #1477
• check for all possible states instead of matching container name with container name in status field by @edcdavid in #1464
• add check for "manifest_list_digest" when looking for certified containers by digest by @edcdavid in #1489
• adding digest in the failure reason of of the affiliated-certification-container-is-certified-digest test by @edcdavid in #1473
• onlinecheck: delete unused function by @jmontesi in #1471

Others

• Bug fix: reduce the TNF image size by @jmontesi in #1449
• Show TC metrics on QE coverage report by @bnshr in #1456
• CNFCERT-612:Cluster-admin permission: Check which subjects have cluster-admin by @aabughosh in #1451
• Avoid printing null error by @rdavid in #1476
• Add 'types' unit tests by @sebrandon1 in #1482
• Delete unused files by @sebrandon1 in #1483

CI Improvements

• Remove 4.0.x YAML by @sebrandon1 in #1432
• Remove 24hr check from QE YAML by @sebrandon1 in #1445
• Remove 'results-html' from 'make test' path by @sebrandon1 in #1457
• Always --pull when building Dockerfile by @sebrandon1 in #1461
• Enable more suites for QE nightly runs by @sebrandon1 in #1447
• Experiment with github-hosted QE runs by @sebrandon1 in #1472
• Remove 'large packages' from disk cleanup by @sebrandon1 in #1478
• Continue on error by @sebrandon1 in #1484
• Free-disk-space update by @sebrandon1 in #1491

Dependency Updates

• Bump gotest.tools/v3 from 3.5.0 to 3.5.1 by @dependabot in #1446
• Bump mfinelli/setup-shfmt from 2 to 3 by @dependabot in #1453
• Upgrade parser to v0.1.2 by @edcdavid in #1452
• Bump github.com/onsi/ginkgo/v2 from 2.12.0 to 2.12.1 by @dependabot in #1460
• Bump ubi8/ubi-minimal from 8.8-1037 to 8.8-1072 by @dependabot in #1459
• Bump ubi8/ubi from 8.8-1032.1692772289 to 8.8-1067 by @dependabot in #1458
• Bump github.com/test-network-function/privileged-daemonset from 1.0.8 to 1.0.10 by @dependabot in #1468
• Bump github.com/test-network-function/test-network-function-claim from 1.0.26 to 1.0.27 by @dependabot in #1467
• Bump helm.sh/helm/v3 from 3.12.3 to 3.13.0 by @dependabot in #1487

Full Changelog: v4.4.0...v4.4.1