Release 2025.1.0

Author: blackduck-serv-builder

Important_Upgrade_Announcement.md

@@ -2,17 +2,15 @@
 
 This repository contains orchestration files and documentation for deploying Black Duck Docker containers.
 
-## Location of Black Duck 2024.10.1 archive:
+## Location of Black Duck 2025.1.0 archive:
 
-https://github.com/blackducksoftware/hub/archive/v2024.10.1.tar.gz
+https://github.com/blackducksoftware/hub/archive/v2025.1.0.tar.gz
 
 NOTE:
 
 Customers upgrading from a version prior to 2018.12.0 will experience a longer than usual upgrade time due to a data migration needed to support new features in
 subsequent releases. Upgrade times will depend on the size of the Black Duck database. If you would like to monitor the process of the upgrade, please contact
-Synopsys Customer Support for instructions.
-
-Customers upgrading from a version prior to 4.2 should contact Synopsys Technical Support for assistance.
+Black Duck Customer Support for instructions.
 
 Customers upgrading from a version prior to 2022.2.0 will have their PostgreSQL data volume automatically migrated from PostgreSQL 9.6.x to PostgreSQL 11.x.
 

README.md

@@ -1,12 +1,12 @@
-# Running Black Duck by Synopsys in Docker (Using Docker Swarm)
+# Running Black Duck in Docker (Using Docker Swarm)
 
 This is the bundle for running with Docker Swarm. 
 
 ## Important Upgrade Announcement
- 
-Customers upgrading from a version prior to 2018.12.0 will experience a longer than usual upgrade time due to a data migration needed to support new features in this release. Upgrade times will depend on the size of the Black Duck database. If you would like to monitor the process of the upgrade, please contact Synopsys Customer Support for instructions.
- 
-Customers upgrading from a version prior to 4.2, should contact Synopsys Technical Support for assistance.
+
+Customers upgrading from a version prior to 2018.12.0 will experience a longer than usual upgrade time due to a data migration needed to support new features in
+this release. Upgrade times will depend on the size of the Black Duck database. If you would like to monitor the process of the upgrade, please contact Black
+Duck Customer Support for instructions.
 
 ## Requirements
 
@@ -27,7 +27,10 @@ The performance of PostgreSQL might degrade if a network volume is used. This ha
 
 ----
 
-Black Duck 2022.2.0 transitions the provided database container from PostgreSQL 9.6.x to PostgreSQL 11.x.  If upgrading from Black Duck 4.2 through Black Duck 2021.10.x to Black Duck 2022.2.0 or later, the data migration is performed automatically when Black Duck 2022.2.0 (or later) is started.  If upgrading from Black Duck versions older than 4.2, please contact Synopsys Technical Support before proceeding.  In either case, Synopsys  _strongly_  recommends backing up the database before upgrading.
+Black Duck 2022.2.0 transitions the provided database container from PostgreSQL 9.6.x to PostgreSQL 11.x. If upgrading from Black Duck 4.2 through Black Duck
+2021.10.x to Black Duck 2022.2.0 or later, the data migration is performed automatically when Black Duck 2022.2.0 (or later) is started. If upgrading from Black
+Duck versions older than 4.2, please contact Black Duck Technical Support before proceeding. In either case, Black Duck  _strongly_  recommends backing up the
+database before upgrading.
 
 ### Migrating before starting Black Duck
 
@@ -138,7 +141,7 @@ docker stack deploy --compose-file docker-compose.yml -c docker-compose.readonly
 
 # Overriding defaults
 
-Sometimes it is necessary to override the defaults settings contained within Black Duck by Synopsys.  In order to perserve 
+Sometimes it is necessary to override the defaults settings contained within Black Duck. In order to preserve
 these from version to version a file called "docker-compose.local-overrides.yml" has been provided.  The sections below 
 describe how to change this file for a variety of circumstances.  In all cases, this file is appended as the last yml file used
 in the docker stack command.  For instance, the "Binary Analysis with External Postgres" command just above would be:

docker-swarm/README.md

@@ -3,7 +3,7 @@
 set -e
 
 TIMEOUT=${TIMEOUT:-10}
-HUB_POSTGRES_VERSION=${HUB_POSTGRES_VERSION:-15-1.9}
+HUB_POSTGRES_VERSION=${HUB_POSTGRES_VERSION:-15-1.10}
 HUB_DATABASE_IMAGE_NAME=${HUB_DATABASE_IMAGE_NAME:-postgres}
 
 function fail() {

docker-swarm/bin/hub_add_replication_user.sh

@@ -5,8 +5,8 @@
 #  2. The database container has been properly initialized.
 
 HUB_DATABASE_IMAGE_NAME=${HUB_DATABASE_IMAGE_NAME:-postgres}
-HUB_POSTGRES_VERSION=${HUB_POSTGRES_VERSION:-15-1.9}
-HUB_VERSION=${HUB_VERSION:-2024.10.1}
+HUB_POSTGRES_VERSION=${HUB_POSTGRES_VERSION:-15-1.10}
+HUB_VERSION=${HUB_VERSION:-2025.1.0}
 OPT_FORCE=
 OPT_LIVE_SYSTEM=
 OPT_MAX_CPU=${MAX_CPU:-1}

docker-swarm/bin/hub_create_data_dump.sh

@@ -14,7 +14,7 @@
 set -o errexit
 
 HUB_DATABASE_IMAGE_NAME=${HUB_DATABASE_IMAGE_NAME:-postgres}
-HUB_POSTGRES_VERSION=${HUB_POSTGRES_VERSION:-15-1.9}
+HUB_POSTGRES_VERSION=${HUB_POSTGRES_VERSION:-15-1.10}
 OPT_MAX_CPU=${MAX_CPU:-1}
 OPT_NO_DATABASE=${NO_DATABASE:-}
 OPT_NO_STORAGE=${NO_STORAGE:-}

docker-swarm/bin/hub_db_migrate.sh

@@ -3,7 +3,7 @@
 set -e
 
 TIMEOUT=${TIMEOUT:-10}
-HUB_POSTGRES_VERSION=${HUB_POSTGRES_VERSION:-15-1.9}
+HUB_POSTGRES_VERSION=${HUB_POSTGRES_VERSION:-15-1.10}
 HUB_DATABASE_IMAGE_NAME=${HUB_DATABASE_IMAGE_NAME:-postgres}
 
 function fail() {

docker-swarm/bin/hub_replication_changepassword.sh

@@ -3,7 +3,7 @@
 set -e
 
 TIMEOUT=${TIMEOUT:-10}
-HUB_POSTGRES_VERSION=${HUB_POSTGRES_VERSION:-15-1.9}
+HUB_POSTGRES_VERSION=${HUB_POSTGRES_VERSION:-15-1.10}
 HUB_DATABASE_IMAGE_NAME=${HUB_DATABASE_IMAGE_NAME:-postgres}
 
 function fail() {

docker-swarm/bin/hub_reportdb_changepassword.sh

@@ -1,14 +1,14 @@
 #!/usr/bin/env bash
 #
-# Copyright (C) 2021 Synopsys Inc.
-# http://www.synopsys.com/
+# Copyright (C) 2021 Black Duck Software, Inc.
+# http://www.blackduck.com/
 # All rights reserved.
 #
 # This software is the confidential and proprietary information of
-# Synopsys ("Confidential Information"). You shall not
+# Black Duck ("Confidential Information"). You shall not
 # disclose such Confidential Information and shall use it only in
 # accordance with the terms of the license agreement you entered into
-# with Synopsys.
+# with Black Duck.
 #
 # Gather system and orchestration data to aide in problem diagnosis.
 # This command should be run by a user with "kubectl" configured.

docker-swarm/bin/kubernetes_check.sh

@@ -1,14 +1,14 @@
 #!/usr/bin/env bash
 #
-# Copyright (C) 2018 Black Duck Software Inc.
-# http://www.blackducksoftware.com/
+# Copyright (C) 2018 Black Duck Software, Inc.
+# http://www.blackduck.com/
 # All rights reserved.
 #
 # This software is the confidential and proprietary information of
-# Black Duck Software ("Confidential Information"). You shall not
+# Black Duck ("Confidential Information"). You shall not
 # disclose such Confidential Information and shall use it only in
 # accordance with the terms of the license agreement you entered into
-# with Black Duck Software.
+# with Black Duck.
 
 # Gather system and orchestration data to aide in problem diagnosis.
 # This command should be run by "root" on the docker host, although
@@ -41,7 +41,7 @@ set -o noglob
 
 readonly NOW="$(date +"%Y%m%dT%H%M%S%z")"
 readonly NOW_ZULU="$(date -u +"%Y%m%dT%H%M%SZ")"
-readonly HUB_VERSION="${HUB_VERSION:-2024.10.1}"
+readonly HUB_VERSION="${HUB_VERSION:-2025.1.0}"
 readonly OUTPUT_FILE="${SYSTEM_CHECK_OUTPUT_FILE:-system_check_${NOW}.txt}"
 readonly PROPERTIES_FILE="${SYSTEM_CHECK_PROPERTIES_FILE:-${OUTPUT_FILE%.txt}.properties}"
 readonly SUMMARY_FILE="${SYSTEM_CHECK_SUMMARY_FILE:-${OUTPUT_FILE%.txt}_summary.properties}"
@@ -326,7 +326,7 @@ readonly WARN="WARNING"
 readonly FAIL="FAIL"
 readonly NOTE="NOTE"
 
-# See https://sig-confluence.internal.synopsys.com/display/SIGBD/Architecture+Overview
+# See https://blackduck.atlassian.net/wiki/spaces/SIGBD/pages/13107681/Architecture+Overview
 declare -ar REPLICABLE=(
     # "SERVICE=status"
     "hub_alert=$WARN"
@@ -1614,6 +1614,7 @@ END
 ################################################################
 # shellcheck disable=SC2155,SC2046
 # shellcheck disable=SC2030,SC2031 # False positives; see https://github.com/koalaman/shellcheck/issues/1409
+# shellcheck disable=SC2319 # allow 'echo "$?"'
 echo_port_status() {
     # shellcheck disable=SC2128 # $FUNCNAME[0] does not work in Alpine ash
     [[ "$#" -eq 1 ]] || error_exit "usage: $FUNCNAME <port>"
@@ -1760,6 +1761,56 @@ EOF
     fi
 }
 
+################################################################
+# Check redis system configuration settings
+#
+# Globals:
+#   SYSCTL_OVERCOMMIT_STATUS -- (out) PASS/FAIL vm.overcommit status message
+#   KERNEL_TRANSPARENT_HUGEPAGES -- (out) PASS/FAIL status message
+# Arguments:
+#   None
+# Returns:
+#   None
+################################################################
+# shellcheck disable=SC2046,SC2155
+check_redis_settings() {
+    if [[ -z "${SYSCTL_OVERCOMMIT_STATUS}" ]] ; then
+        if ! is_linux ; then
+            readonly SYSCTL_OVERCOMMIT_STATUS="$UNKNOWN -- non-linux system"
+            return
+        fi
+
+        if ! have_command sysctl ; then
+            readonly SYSCTL_OVERCOMMIT_STATUS="$UNKNOWN -- sysctl not found"
+        elif is_macos ; then
+            readonly SYSCTL_OVERCOMMIT_STATUS="$PASS -- not applicable to macOS"
+        else
+            echo "Checking sysctl vm.overcommit_memory setting..."
+            local overcommit=$(sysctl vm.overcommit_memory | awk -F' = ' '{print $2}')
+            if [[ "$overcommit" -ne 1 ]]; then
+                readonly SYSCTL_OVERCOMMIT_STATUS="$FAIL: vm.overcommit_memory = ${overcommit}. Redis may fail under low memory conditions. See https://redis.io/docs/latest/develop/get-started/faq/#background-saving-fails-with-a-fork-error-on-linux"
+            else
+                readonly SYSCTL_OVERCOMMIT_STATUS="$PASS: vm.overcommit_memory = ${overcommit}"
+            fi
+        fi
+
+        if [[ -r /sys/kernel/mm/transparent_hugepage/enabled ]] ; then
+            echo "Checking transparent hugepage setting..."
+            local kth=$(cat /sys/kernel/mm/transparent_hugepage/enabled)
+            if [[ "$kth" =~ \[always\] ]] || [[ "$kth" == 'always' ]] ; then
+                KERNEL_TRANSPARENT_HUGEPAGES="$FAIL: '$kth'. See the 'Kernel Memory' section of the redis tuning guide."
+            else
+                KERNEL_TRANSPARENT_HUGEPAGES="$PASS: '$kth'"
+            fi
+        else
+            KERNEL_TRANSPARENT_HUGEPAGES="$UNKNOWN -- /sys/kernel/mm/transparent_hugepage/enabled not found"
+        fi
+
+        # https://redis.io/learn/operate/redis-at-scale/talking-to-redis/initial-tuning has many
+        # more setting suggestions that we are not checking...
+    fi
+}
+
 ################################################################
 # Report login manager settings
 #
@@ -2740,7 +2791,7 @@ check_container_memory() {
 # Globals:
 #   RUNNING_HUB_VERSION -- (out) running Black Duck version.
 #   RUNNING_BDBA_VERSION -- (out) running BDBA version.
-#   RUNNING_ALERT_VERSION -- (out) running Synopsys Alert version.
+#   RUNNING_ALERT_VERSION -- (out) running Black Duck Alert version.
 #   RUNNING_OTHER_VERSIONS -- (out) other Black Duck product versions.
 #   RUNNING_VERSION_STATUS -- (out) pass/fail version check message.
 # Arguments:
@@ -3787,7 +3838,7 @@ check_reg_server_reachable() {
 }
 
 ################################################################
-# Test connectivity with the Synopsys artifactory
+# Test connectivity with the Black Duck artifactory
 #
 # Globals: (set indirectly)
 #   SIG_REPO_RESOLVE_RESULT, SIG_REPO_RESOLVE_OUTPUT,
@@ -3806,7 +3857,7 @@ check_sig_repo_reachable() {
             return 0
         fi
 
-        local -r SIG_REPO_HOST="sig-repo.synopsys.com"
+        local -r SIG_REPO_HOST="repo.blackduck.com"
         local -r SIG_REPO_URL="https://${SIG_REPO_HOST}/"
         tracepath_host "${SIG_REPO_HOST}" "SIG_REPO"
         probe_url "${SIG_REPO_URL}" "SIG_REPO" "${SIG_REPO_URL}"
@@ -4587,6 +4638,18 @@ IPVS timeout check ${IPVS_TIMEOUT_STATUS}
 
 ${TCP_KEEPALIVE_TIMEOUT_DESC}
 
+$(generate_report_section "Redis Initial Configuration")
+
+Memory overcommit: ${SYSCTL_OVERCOMMIT_STATUS}
+
+Kernel transparent hugepages: ${KERNEL_TRANSPARENT_HUGEPAGES}
+
+See https://redis.io/learn/operate/redis-at-scale/talking-to-redis/initial-tuning and
+https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/latency/ for
+other tuning suggestions.
+
+Make sure to configure settings on all nodes! This script only checks the master.
+
 $(generate_report_section "Login manager settings")
 
 Login manager settings: ${LOGINCTL_STATUS}
@@ -4723,7 +4786,7 @@ ${REG_TRACEPATH_RESULT}
 Web access to Black Duck registration service via docker containers:
 ${REG_CONTAINER_WEB_REPORT}
 
-$(generate_report_section "Synopsys artifactory connectivity")
+$(generate_report_section "Black Duck artifactory connectivity")
 
 ${SIG_REPO_URL_REACHABLE}
 
@@ -4733,7 +4796,7 @@ ${SIG_REPO_RESOLVE_OUTPUT}
 Path information: ${SIG_REPO_TRACEPATH_CMD}
 ${SIG_REPO_TRACEPATH_RESULT}
 
-Web access to Synopsys artifactory via docker containers:
+Web access to Black Duck artifactory via docker containers:
 ${SIG_REPO_CONTAINER_WEB_REPORT}
 
 $(generate_report_section "Black Duck Docker registry connectivity")
@@ -5043,6 +5106,7 @@ main() {
 
     check_hyperthreading
     check_iosched
+    check_redis_settings
 
     check_entropy
     get_interface_info

docker-swarm/bin/system_check.sh

@@ -1,13 +1,13 @@
 #!/usr/bin/env perl
 #
-# Copyright (C) 2023 Synopsys Inc.
-# https://www.synopsys.com/
+# Copyright (C) 2023 Black Duck Software, Inc.
+# https://www.blackduck.com/
 # All rights reserved.
 #
 # This software is the confidential and proprietary information of
-# Synopsys ("Confidential Information"). You shall not disclose such
+# Black Duck ("Confidential Information"). You shall not disclose such
 # Confidential Information and shall use it only in accordance with
-# the terms of the license agreement you entered into with Synopsys.
+# the terms of the license agreement you entered into with Black Duck.
 
 # Migrate sources from pre-2024.1.0 upload-cache storage volumes to a
 # 2024.1.0 or later server.  Note that uploaded source is still
@@ -55,7 +55,8 @@
 my $TOKEN_EXPIRATION=0;
 
 # Parse the command line
-my $verbose = 1;
+my $verbose = 0;
+my $continue = 0;
 my $help = 0;
 my $dry_run = 0;
 my $insecure = 0;
@@ -69,6 +70,7 @@
            'seal-key-path=s' => \$SEAL_KEY_PATH,
            'keys-volume=s' => \$KEYS_VOLUME,
            'data-volume=s' => \$DATA_VOLUME,
+           'continue' => \$continue,
            'verbose|v+' => \$verbose,
            'dry-run|n' => \$dry_run,
            'insecure|k' => \$insecure,
@@ -101,9 +103,8 @@
     $filenum ++;
     if ($file =~ m:/[a-f0-9]{40}$:) {
         print "... $file ($filenum of ${\scalar(@files)})\n" if ($verbose > 0);
-        my $rawData = &get_bytes($file);
-        my $text = &decrypt($file, $masterKey, $rawData);
-        &upload_source($file, $text);
+        eval { &process_file($file) };
+        if ($@) { if ($continue) { warn $@ } else { die $@ } };
     }
 }
 
@@ -126,8 +127,9 @@ sub usage {
     --keys-volume <path>    Location of the upload-cache keys volume [$KEYS_VOLUME]
     --data-volume <path>    Location of the upload-cache data volume [$DATA_VOLUME]
     --insecure | -k         Skip SSL certificate verification
-    --dry-run | -n          Do everything _except_ upload files
-    --verbose               Print more output, repeatable, alias '-v'
+    --continue              Continue processing files after an error
+    --dry-run | -n          Do everything except upload data
+    --verbose | -v          Print more output, repeatable
 
 Values can be supplied on the command line via options or via environment variables
 (URL, API_TOKEN, SEAL_KEY_PATH, KEYS_VOLUME, or DATA_VOLUME).
@@ -224,3 +226,12 @@ sub upload_source {
                                 Content          => $data);
     die "*** Unable to upload $file: @{[ $response->status_line ]}\n" unless $response->is_success;
 }
+
+# Process a single file
+sub process_file {
+    my ($file) = @_;
+
+    my $rawData = &get_bytes($file);
+    my $text = &decrypt($file, $masterKey, $rawData);
+    &upload_source($file, $text);  
+}