Enable automation of the CodeQL CLI upgrade process

.github/workflows/upgrade_codeql_dependencies.yml

@@ -18,10 +18,20 @@
     runs-on: ubuntu-22.04
     steps:
       - name: Checkout
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
+
+      - name: Fetch CodeQL
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
+          RUNNER_TEMP: ${{ runner.temp }}
+        run: |
+          cd $RUNNER_TEMP
+          gh release download "v${CODEQL_CLI_VERSION}" --repo https://github.com/github/codeql-cli-binaries --pattern codeql-linux64.zip
+          unzip -q codeql-linux64.zip
+          echo "$RUNNER_TEMP/codeql/" >> $GITHUB_PATH
 
       - name: Install Python
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: "3.9"
 
@@ -35,27 +45,27 @@
         run: |
           python3 scripts/upgrade-codeql-dependencies/upgrade-codeql-dependencies.py --cli-version "$CODEQL_CLI_VERSION"
 
-      - name: Fetch CodeQL
-        env:
-          GITHUB_TOKEN: ${{ github.token }}
-          RUNNER_TEMP: ${{ runner.temp }}
-        run: |
-          cd $RUNNER_TEMP
-          gh release download "v${CODEQL_CLI_VERSION}" --repo https://github.com/github/codeql-cli-binaries --pattern codeql-linux64.zip
-          unzip -q codeql-linux64.zip
-
       - name: Update CodeQL formatting based on new CLI version
         env:
           RUNNER_TEMP: ${{ runner.temp }}
         run: |
-          find cpp \( -name '*.ql' -or -name '*.qll' \) -print0 | xargs -0 --max-procs "$XARGS_MAX_PROCS" $RUNNER_TEMP/codeql/codeql query format --in-place
-          find c \( -name '*.ql' -or -name '*.qll' \) -print0 | xargs -0 --max-procs "$XARGS_MAX_PROCS" $RUNNER_TEMP/codeql/codeql query format --in-place
+          find cpp \( -name '*.ql' -or -name '*.qll' \) -print0 | xargs -0 --max-procs "$XARGS_MAX_PROCS" codeql query format --in-place
+          find c \( -name '*.ql' -or -name '*.qll' \) -print0 | xargs -0 --max-procs "$XARGS_MAX_PROCS" codeql query format --in-place
 
       - name: Create Pull Request
-        uses: peter-evans/create-pull-request@v3
+        uses: peter-evans/create-pull-request@v7
         with:
-          title: "Upgrading `github/codeql` dependency to ${{ github.event.inputs.codeql_cli_version }}"
-          body: "This PR upgrades the CodeQL CLI version to ${{ github.event.inputs.codeql_cli_version }}."
+          title: "Upgrade `github/codeql` dependency to ${{ github.event.inputs.codeql_cli_version }}"
+          body: |
+            This PR upgrades the CodeQL CLI version to ${{ github.event.inputs.codeql_cli_version }}.
+
+            ## CodeQL dependency upgrade checklist:
+
+            - [ ] Confirm the code has been correctly reformatted according to the new CodeQL CLI.
+            - [ ] Identify any CodeQL compiler warnings and errors, and update queries as required.
+            - [ ] Validate that the `github/codeql` test cases succeed.
+            - [ ] Address any CodeQL test failures in the `github/codeql-coding-standards` repository.
+            - [ ] Validate performance vs pre-upgrade, using /test-performance
           commit-message: "Upgrading `github/codeql` dependency to ${{ github.event.inputs.codeql_cli_version }}"
           delete-branch: true
           branch: "codeql/upgrade-to-${{ github.event.inputs.codeql_cli_version }}"

docs/development_handbook.md

@@ -496,46 +496,53 @@ There are two external dependencies required for running the coding standards qu
 
 For the purpose of this repository, and any tool qualification, we consider these external dependencies to be "black boxes" which require verification when upgrading.
 
-To (a) clearly specify the supported versions of these external dependencies and to (b) enable automation around them, the repository contains a `supported_codeql_configs.json` which lists the sets of supported configurations. There are four fields:
+To (a) clearly specify the supported versions of these external dependencies and to (b) enable automation around them, the repository contains a `supported_codeql_configs.json` which lists the sets of supported configurations under the `supported_environments` property. There are three fields:
 
 - `codeql_cli` - this is the plain version number of the supported CodeQL CLI, e.g. `2.6.3`.
 - `codeql_standard_library` - this is the name of a tag on the `github.com/github/codeql` repository. The tag should be compatible with the CodeQL CLI given above. This would typically use the `codeql-cli/v<version-number>` tag for the release, although any tag which is compatible is allowed.
 - `codeql_cli_bundle` - (optional) - if present, describes the CodeQL CLI bundle version that is compatible. The bundle should include precisely the CodeQL CLI version and CodeQL Standard Library versions specified in the two mandatory fields.
-- `ghes` - (optional) - if present describes the GitHub Enterprise Server release whose integrated copy of the CodeQL Action points to the CodeQL CLI bundle specified in the `codeql_cli_bundle` field.
 
 #### Upgrading external dependencies
 
 To upgrade the CodeQL external dependencies:
 
 1. Determine appropriate versions of the CodeQL CLI and `github/codeql` repository, according to the release schedule and customer demands.
 2. Determine if there is a compatible CodeQL CLI bundle version by looking at the releases specified at [CodeQL Action releases](https://github.com/github/codeql-action/releases). The bundle always includes the standard library at the version specified by the `codeql-cli/v<version-number>` tag in the `github/codeql` repository.
-3. If you find a compatible CodeQL CLI bundle, determine whether that bundle was released in a GitHub Enterprise server release, by inspecting the `defaults.json` file at https://github.com/github/codeql-action/blob/main/lib/defaults.json#L2 for the CodeQL Action submitted with
-4. Populated the `supported_codeql_configs.json` file with the given values, ensuring to delete the optional fields if they are not populated.
-5. Submit a Pull Request to the `github/codeql-coding-standards` repository with the title `Upgrade `github/codeql` dependency to <insert codeql_standard_library value>`. Use this template for the description, filling :
 
-    ```md
-    This PR updates the `supported_codeql_configs.json` file to target:
+If all components are being upgraded to a consistent veresion (e.g. CodeQL CLI v2.15.5, with `github/codeql` tag `codeql-cli/v2.15.5` and bundle `codeql-cli-bundle-v2.15.5`) then the following process can be used:
+
+1. Run the [upgrade_codeql_dependencies.yml](./github/workflows/upgrade_codeql_dependencies.yml) workflow, with the plain version number, e.g. `2.15.5`. This will:
+  - Download the specified version of the CodeQL CLI
+  - Run the [upgrade-codeql-dependencies.py](scripts/release/upgrade-codeql-dependencies.py) script, which
+    - Validates the version selected exists in all relevant places
+    - Updates the `supported_codeql_configs.json` file.
+    - Updates each `qlpack.yml` in the repository with an appropriate value for the `codeql/cpp-all` pack, consistent with the selected CodeQL CLI version.
+    - Updates each `codeql-lock.yml` file to upgrade to the new version.
+2. Follow the dependency upgrade checklist, confirming each step. The `.github/workflows/standard_library_upgrade_tests.yml` will trigger automation for running the `github/codeql` unit tests with the appropriate CLI version.
+3. Once all the automate tests have passed, and the checklist is complete, the PR can be merged.
+4. An internal notification should be shared with the development team.
 
-    - CodeQL CLI <codeql_cli>
-    - CodeQL Standard Library <codeql_standard_library>
-    - GHES <ghes>
-    - CodeQL CLI Bundle <date_of_bundle>
+If the upgrade is of mismatched versions you will need to manually create the upgrade following this process:
 
-    <EITHER:This should match the versions of CodeQL deployed with GitHub Enterprise Server <ghes>>
-    <OR: This does not match any released version of GitHub Enterprise Server.>
+1. Populate the `supported_codeql_configs.json` file with the given values, ensuring to delete the optional fields if they are not populated.
+2. Submit a Pull Request to the `github/codeql-coding-standards` repository with the title `Upgrade `github/codeql` dependency to <insert codeql_standard_library value>`. Use this template for the description, filling:
+
+    ```md
+    This PR updates the `supported_codeql_configs.json` file to target CodeQL CLI <codeql_cli>.
 
     ## CodeQL dependency upgrade checklist:
 
-    - [ ] Reformat our CodeQL using the latest version (if required)
+    - [ ] Confirm the code has been correctly reformatted according to the new CodeQL CLI.
     - [ ] Identify any CodeQL compiler warnings and errors, and update queries as required.
     - [ ] Validate that the `github/codeql` test cases succeed.
     - [ ] Address any CodeQL test failures in the `github/codeql-coding-standards` repository.
-    - [ ] Validate performance vs pre-upgrade
+    - [ ] Validate performance vs pre-upgrade, using /test-performance
     ```
 
-6. Follow the dependency upgrade checklist, confirming each step. The `.github/workflows/standard_library_upgrade_tests.yml` will trigger automation for running the `github/codeql` unit tests with the appropriate CLI version.
-7. Once all the automate tests have passed, and the checklist is complete, the PR can be merged.
-8. An internal notification should be shared with the development team.
+3. Follow the dependency upgrade checklist, confirming each step. The `.github/workflows/standard_library_upgrade_tests.yml` will trigger automation for running the `github/codeql` unit tests with the appropriate CLI version.
+4. Once all the automate tests have passed, and the checklist is complete, the PR can be merged.
+5. An internal notification should be shared with the development team.
+
 
 ### Release process
 

scripts/upgrade-codeql-dependencies/requirements.txt

@@ -4,3 +4,4 @@ idna==3.4
 requests==2.31.0
 semantic-version==2.10.0
 urllib3==1.26.18
+pyyaml==6.0.1

scripts/upgrade-codeql-dependencies/upgrade-codeql-dependencies.py

@@ -1,18 +1,23 @@
 import json
 import requests
-from typing import Optional, Dict, List
+from typing import Optional, Dict, List, Tuple
 from semantic_version import Version
 from pathlib import Path
+import yaml
 
 SCRIPT_PATH = Path(__file__)
-SUPPORTED_VERSIONS_PATH = SCRIPT_PATH.parent.parent.parent / "supported_codeql_configs.json"
+CODING_STANDARDS_ROOT = SCRIPT_PATH.parent.parent.parent
+SUPPORTED_VERSIONS_PATH = CODING_STANDARDS_ROOT / "supported_codeql_configs.json"
 
-def get_compatible_stdlib(version: Version) -> Optional[str]:
+def get_compatible_stdlib(version: Version) -> Optional[Tuple[str, str]]:
     tag = f"codeql-cli/v{version}"
     response = requests.get(f"https://raw.githubusercontent.com/github/codeql/{tag}/cpp/ql/lib/qlpack.yml")
 
     if response.status_code == 200:
-        return tag
+        # Parse the qlpack.yml returned in the response as a yaml file to read the version property
+        qlpack = yaml.safe_load(response.text)
+        if qlpack is not None and "version" in qlpack:
+            return (tag, qlpack["version"])
     return None
 
 def get_compatible_bundle(version: Version, token: str) -> Optional[str]:
@@ -30,15 +35,17 @@ def get_compatible_bundle(version: Version, token: str) -> Optional[str]:
 def main(cli_version : str, github_token: str) -> None:
     try:
         parsed_cli_version = Version(cli_version)
-        compatible_stdlib = get_compatible_stdlib(parsed_cli_version)
-        if compatible_stdlib is None:
+        compatible_stdlib_return = get_compatible_stdlib(parsed_cli_version)
+        if compatible_stdlib_return is None:
             print(f"Unable to find compatible standard library for: {parsed_cli_version}")
             exit(1)
         compatible_bundle = get_compatible_bundle(parsed_cli_version, github_token)
         if compatible_bundle is None:
             print(f"Unable to find compatible bundle for: {parsed_cli_version}")
             exit(1)
 
+        compatible_stdlib_tag, compatible_stdlib_version = compatible_stdlib_return
+
         with SUPPORTED_VERSIONS_PATH.open("r") as f:
             supported_versions = json.load(f)
 
@@ -49,10 +56,36 @@ def main(cli_version : str, github_token: str) -> None:
         supported_env = supported_envs[0]
         supported_env["codeql_cli"] = str(parsed_cli_version)
         supported_env["codeql_cli_bundle"] = compatible_bundle
-        supported_env["codeql_standard_library"] = compatible_stdlib
+        supported_env["codeql_standard_library"] = compatible_stdlib_tag
 
         with SUPPORTED_VERSIONS_PATH.open("w") as f:
             json.dump(supported_versions, f, indent=2)
+
+        # Find every qlpack.yml file in the repository
+        qlpack_files = list(CODING_STANDARDS_ROOT.rglob("qlpack.yml"))
+        # Filter out any files that are in a hidden directory
+        qlpack_files = [f for f in qlpack_files if not any(part for part in f.parts if part.startswith("."))]
+
+        # Update the "codeql/cpp-all" entries in the "dependencies" property in every qlpack.yml file
+        updated_qlpacks = []
+        for qlpack_file in qlpack_files:
+            with qlpack_file.open("r") as f:
+                qlpack = yaml.safe_load(f)
+            print("Updating dependencies in " + str(qlpack_file))
+            if "codeql/cpp-all" in qlpack["dependencies"]:
+                qlpack["dependencies"]["codeql/cpp-all"] = compatible_stdlib_version
+                with qlpack_file.open("w") as f:
+                    yaml.safe_dump(qlpack, f, sort_keys=False)
+                updated_qlpacks.append(qlpack_file.parent)
+
+        # Call CodeQL to update the lock files by running codeql pack upgrade
+        # Note: we need to do this after updating all the qlpack files,
+        #       otherwise we may get dependency resolution errors
+        for qlpack in updated_qlpacks:
+            print("Updating lock files for " + str(qlpack))
+            os.system(f"codeql pack upgrade {qlpack}")
+
+
     except ValueError as e:
         print(e)
         exit(1)