Does not detect use of CodeQL (SAST)

[mwager]

Describe the bug
This repo is using CodeQL for SAST scanning:

https://github.com/secureIO-GmbH/VulnerabilityHeatmap/actions/workflows/github-code-scanning/codeql

Reproduction steps
Steps to reproduce the behavior:
1.

2. Does not detect SAST:
   

Expected behavior
Docs say:

Maybe it is missing to detect the direct actions? (without a workflow file?)

Am I doing smt wrong?

[spencerschrock]

Maybe it is missing to detect the direct actions? (without a workflow file?)

The CodeQL detection can work without a workflow file, depending on if it still runs the scan on the PR.

Right now the code:

1. looks for the last 30 commits
2. looks for the PR associated with the commit
3. checks if the PR has CodeQL checks before merge

It seems like the CodeQL runs in your repo, take place once the commits are pushed to main. Which has been discussed before: #3451 #1580. We should probably re-evaluate our notion on pre/post merge.

[mwager]

Ok thanks for the explanation, strange thing is there are PRs, example: secureIO-GmbH/VulnerabilityHeatmap#11

If the other issues already address this, feel free to close here.

[spencerschrock]

Ok thanks for the explanation, strange thing is there are PRs, example: secureIO-GmbH/VulnerabilityHeatmap#11

Specifically it evaluates the last 30 commits in main, and the PRs which they came from. Since that PR is still open, it's not being considered.

[thepwagner]

For GitHub+CodeQL specifically, I think scorecard could also consider this API endpoint: https://docs.github.com/en/rest/code-scanning/code-scanning?apiVersion=2022-11-28#get-a-code-scanning-default-setup-configuration--status-codes

The downside is that it needs administration:read permission on the repository.

Disabled:

{
  "state": "not-configured",
  "languages": [

  ],
  "query_suite": "default",
  "updated_at": null,
  "schedule": null
}

Enabled:

{
  "state": "configured",
  "languages": [
    "ruby"
  ],
  "query_suite": "default",
  "updated_at": "2024-04-15T14:22:53Z",
  "schedule": "weekly"
}

[jhutchings1]

As @thepwagner mentioned, default setup is the new standard way that developers enable CodeQL scanning. It's true that the API for accessing that configuration requires administrative privileges, but there's actually a workaround that I think would make tons of sense for Scorecard and should be a relatively straightforward swap.

GitHub also has an API that lists Actions workflows that have run on the repository. This includes dynamic workflows like the default setup workflow for CodeQL. Here's an example from a test repo of mine, which lists both the CodeQL and Dependabot updates workflows (slightly redacted). The path for those workflows is consistent and you can count on that as an indicator that the analysis is enabled.

{
  "total_count": 2,
  "workflows": [
    {
      "id": ...,
      "node_id": "...",
      "name": "Dependabot Updates",
      "path": "dynamic/dependabot/dependabot-updates",
      "state": "active",
      "created_at": "2024-05-01T10:10:17.000-07:00",
      "updated_at": "2024-05-01T10:10:17.000-07:00",
      "url": "...",
      "html_url": "...",
      "badge_url": "..."
    },
    {
      "id": ...,
      "node_id": "...",
      "name": "CodeQL",
      "path": "dynamic/github-code-scanning/codeql",
      "state": "active",
      "created_at": "2024-01-23T09:14:42.000-08:00",
      "updated_at": "2024-01-23T09:14:42.000-08:00",
      "url": "...",
      "html_url": "...",
      "badge_url": "..."
    }
  ]
}

[gcanlin]

Hi. Is anyone looking into this issue? As @jhutchings1 said, we can use the GitHub API for listing Actions workflows to enhance the detection of SAST. Just match the name or path with string "codeQL" and so on.