Skip to content

Commit 027365c

Browse files
committed
Update the analyze databases article
This change updates the analyze databases article to clarify examples. It reorganizes to put packs examples first and rearranges a few paragraphs.
1 parent 3165bab commit 027365c

File tree

1 file changed

+95
-94
lines changed

1 file changed

+95
-94
lines changed

docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst

Lines changed: 95 additions & 94 deletions
Original file line numberDiff line numberDiff line change
@@ -15,11 +15,15 @@ For information about writing queries to run with ``database analyze``, see
1515

1616
Before starting an analysis you must:
1717

18-
- :doc:`Set up the CodeQL CLI <getting-started-with-the-codeql-cli>` so that it can find the queries
19-
and libraries included in the CodeQL repository.
18+
- :doc:`Set up the CodeQL CLI <getting-started-with-the-codeql-cli>` to run commands locally and
19+
optionally check out the CodeQL repository if you want direct access to the CodeQL core queries
20+
and libraries.
2021
- :doc:`Create a CodeQL database <creating-codeql-databases>` for the source
2122
code you want to analyze.
2223

24+
It is simplest to run ``database analyze`` using CodeQL packs. You can also
25+
run the command using queries from a local checkout of the CodeQL repository,
26+
which you may want to do if you want to customize the CodeQL core queries.
2327

2428
Running ``codeql database analyze``
2529
------------------------------------
@@ -34,7 +38,7 @@ When you run ``database analyze``, it:
3438

3539
You can analyze a database by running the following command::
3640

37-
codeql database analyze <database> --format=<format> --output=<output> <queries>
41+
codeql database analyze <database> --format=<format> --output=<output> ...<query-specifiers>
3842

3943

4044
You must specify:
@@ -52,7 +56,7 @@ You must specify:
5256

5357
You can also specify:
5458

55-
- ``...<query-specifications>``: a list of queries to run over your database. This
59+
- ``...<query-specifiers>``: a list of queries to run over your database. This
5660
is a list of arguments. Where each argument can be:
5761

5862
- a path to a query file
@@ -62,7 +66,7 @@ You can also specify:
6266
- with an optional version range
6367
- with an optional path to a query, directory, or query suite inside the pack
6468

65-
If omitted, the default query suite for the language of the database being analyzed will be used. For more information, see the :ref:`examples <database-analyze-examples>` below.
69+
If omitted, the default query suite for the language of the database being analyzed will be used. For the complete syntax of query specifiers, see the :ref:`examples <specifying-which-queries>` below.
6670

6771
- ``--sarif-category``: an identifying category for the results. Used when
6872
you want to upload more than one set of results for a commit.
@@ -97,53 +101,92 @@ the `database analyze reference documentation <../manual/database-analyze>`__.
97101

98102
.. _database-analyze-examples:
99103

100-
Examples
101-
--------
104+
.. _specifying-which-queries:
102105

103-
The following examples assume your CodeQL databases have been created in a
104-
directory that is a sibling of your local copies of the CodeQL and CodeQL for Go
105-
repositories.
106+
Specifying which queries to run in a CodeQL pack
107+
------------------------------------------------
106108

107-
Running a single query
108-
~~~~~~~~~~~~~~~~~~~~~~
109+
*Query specifiers* are used by ``codeql database analyze`` and other commands that operate on a set of queries.
110+
The complete way to specify a set of queries is in the form ``scope/name@range:path``, where:
109111

110-
To run a single query over a CodeQL database for a JavaScript codebase,
111-
you could use the following command from the directory containing your database::
112+
- ``scope/name`` is the qualified name of a CodeQL pack.
113+
- ``range`` is a `semver range <https://docs.npmjs.com/cli/v6/using-npm/semver#ranges>`_.
114+
- ``path`` is a file system path to a single query, a directory containing queries, or a query suite file.
112115

113-
codeql database analyze <javascript-database> ../ql/javascript/ql/src/Declarations/UnusedVariable.ql --format=csv --output=js-analysis/js-results.csv
116+
When you specify a ``scope/name``, the ``range`` and ``path`` are
117+
optional. If you omit a ``range`` then the latest version of the
118+
specified pack is used. If you omit a ``path`` then the default query suite
119+
of the specified pack is used.
114120

115-
This command runs a simple query that finds potential bugs related to unused
116-
variables, imports, functions, or classes---it is one of the JavaScript
117-
queries included in the CodeQL repository. You could run more than one query by
118-
specifying a space-separated list of similar paths.
121+
The ``path`` can be one of a ``.ql`` query file, a directory
122+
containing one or more queries, or a ``.qls`` query suite file. If
123+
you omit a pack name, then you must provide a ``path``,
124+
which will be interpreted relative to the working directory
125+
of the current process. Glob patterns are not supported.
119126

120-
The analysis generates a CSV file (``js-results.csv``) in a new directory
121-
(``js-analysis``).
127+
If you specify a ``scope/name`` and ``path``, then the ``path`` cannot
128+
be absolute. It is considered relative to the root of the CodeQL
129+
pack.
122130

123-
You can also run your own custom queries with the ``database analyze`` command.
124-
For more information about preparing your queries to use with the CodeQL CLI,
125-
see ":doc:`Using custom queries with the CodeQL CLI <using-custom-queries-with-the-codeql-cli>`."
131+
Example query specifiers
132+
------------------------
126133

127-
If you do not have the CodeQL repository checked out, you can execute the same queries by specifying the query pack name and the path to the queries::
134+
* ``codeql/python-queries`` - All the queries in the default query suite of the latest version of the ``codeql/python-queries`` pack.
135+
* ``codeql/python-queries@1.2.3`` - All the queries in the default query suite of version ``1.2.3`` of the ``codeql/python-queries`` pack.
136+
* ``codeql/python-queries@~1.2.3`` - All the queries in the default query suite of the latest version of the ``codeql/python-queries`` pack that is >= ``1.2.3`` and < ``1.3.0``.
137+
* ``codeql/python-queries:Functions`` - All queries in the ``Functions`` directory in the latest version of the ``codeql/python-queries`` pack.
138+
* ``codeql/python-queries@1.2.3:Functions`` - All queries in the ``Functions`` directory in version 1.2.3 of the ``codeql/python-queries`` pack.
139+
* ``codeql/python-queries@1.2.3:codeql-suites/python-code-scanning.qls`` - All queries in the ``codeql-suites/python-code-scanning.qls`` directory in version 1.2.3 of the ``codeql/python-queries`` pack.
140+
* ``suites/my-suite.qls`` - All queries in the ``suites/my-suite.qls`` file relative to the current working directory.
128141

129-
codeql database analyze --download <javascript-database> codeql/javascript-queries:Declarations/UnusedVariable.ql --format=csv --output=js-analysis/js-results.csv
142+
Examples of using ``codeql database analyze``
143+
---------------------------------------------
130144

131-
Use the ``--download`` flag to download the query pack if it isn't yet available locally.
145+
The following examples show how to run ``database analyze`` using CodeQL packs as
146+
well as using a local checkout of the CodeQL repository. These examples assume your
147+
CodeQL databases have been created in a directory that is a sibling of your local
148+
copies of the CodeQL repository.
132149

133150
.. _run-query-pack:
134151

135-
Running a CodeQL pack
136-
~~~~~~~~~~~~~~~~~~~~~
152+
Running a CodeQL query pack
153+
~~~~~~~~~~~~~~~~~~~~~~~~~~~
137154

138155
.. include:: ../reusables/beta-note-package-management.rst
139156

140157
To run an existing CodeQL query pack from the GitHub Container registry, you can specify one or more
141-
pack names and use the ``--download`` flag::
158+
pack names::
142159

143160
codeql database analyze <database> microsoft/coding-standards@1.0.0 github/security-queries --format=sarifv2.1.0 --output=query-results.sarif --download
144161

145-
The ``analyze`` command above runs the default suite from ``microsoft/coding-standards v1.0.0`` and the latest version of ``github/security-queries`` on the specified database.
146-
For further information about default suites, see ":ref:`Publishing and using CodeQL packs <publishing-and-using-codeql-packs>`".
162+
The ``analyze`` command above runs the default query suite of two CodeQL query packs: ``microsoft/coding-standards`` version 1.0.0 and the latest version of ``github/security-queries`` on the
163+
specified database. For further information about default suites, see
164+
":ref:`Publishing and using CodeQL packs <publishing-and-using-codeql-packs>`".
165+
166+
The ``--download`` flag is optional. Using it will ensure the query pack is downloaded if it isn't yet available locally.
167+
168+
Running a single query
169+
~~~~~~~~~~~~~~~~~~~~~~
170+
171+
To run a single query over a CodeQL database for a JavaScript codebase,
172+
you could use the following command from the directory containing your database::
173+
174+
codeql database analyze --download <javascript-database> codeql/javascript-queries:Declarations/UnusedVariable.ql --format=csv --output=js-analysis/js-results.csv
175+
176+
This command runs a simple query that finds potential bugs related to unused
177+
variables, imports, functions, or classes---it is one of the JavaScript
178+
queries included in the CodeQL repository. You could run more than one query by
179+
specifying a space-separated list of similar paths.
180+
181+
The analysis generates a CSV file (``js-results.csv``) in a new directory (``js-analysis``).
182+
183+
If you have the CodeQL repository checked out, you can execute the same queries by specifying the path to the query directly::
184+
185+
codeql database analyze <javascript-database> ../ql/javascript/ql/src/Declarations/UnusedVariable.ql --format=csv --output=js-analysis/js-results.csv
186+
187+
You can also run your own custom queries with the ``database analyze`` command.
188+
For more information about preparing your queries to use with the CodeQL CLI,
189+
see ":doc:`Using custom queries with the CodeQL CLI <using-custom-queries-with-the-codeql-cli>`."
147190

148191
Running all queries in a directory
149192
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -163,89 +206,47 @@ recursively, so any queries contained in subfolders will also be executed.
163206
pack's default queries in the analysis, or run one of the
164207
code scanning query suites.
165208

166-
For example, to execute all Python queries contained in the ``Functions`` directory you would run::
209+
For example, to execute all Python queries contained in the ``Functions`` directory in the
210+
``codeql/python-queries`` query pack you would run::
211+
212+
codeql database analyze <python-database> codeql/python-queries:Functions --format=sarif-latest --output=python-analysis/python-results.sarif --download
213+
214+
Alternatively, if you have the CodeQL repository checked out, you can execute the
215+
same queries by specifying the path to the directory directly::
167216

168217
codeql database analyze <python-database> ../ql/python/ql/src/Functions/ --format=sarif-latest --output=python-analysis/python-results.sarif
169218

170219
When the analysis has finished, a SARIF results file is generated. Specifying ``--format=sarif-latest`` ensures
171220
that the results are formatted according to the most recent SARIF specification
172221
supported by CodeQL.
173222

174-
.. _including-query-help-for-custom-codeql-queries-in-sarif-files:
175-
176-
Running a subset of queries in a CodeQL pack
177-
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
178-
179-
If you are using CodeQL CLI v2.8.1 or later, you can include a path at the end of a pack specification to run a subset of queries inside the pack. This applies to any command that locates or runs queries within a pack.
180-
181-
The complete way to specify a set of queries is in the form ``scope/name@range:path``, where:
182-
183-
- ``scope/name`` is the qualified name of a CodeQL pack.
184-
- ``range`` is a `semver range <https://docs.npmjs.com/cli/v6/using-npm/semver#ranges>`_.
185-
- ``path`` is a file system path to a single query, a directory containing queries, or a query suite file.
186-
187-
When you specify a ``scope/name``, the ``range`` and ``path`` are
188-
optional. If you omit a ``range`` then the latest version of the
189-
specified pack is used. If you omit a ``path`` then the default query suite
190-
of the specified pack is used.
191-
192-
The ``path`` can be one of a ``*.ql`` query file, a directory
193-
containing one or more queries, or a ``.qls`` query suite file. If
194-
you omit a pack name, then you must provide a ``path``,
195-
which will be interpreted relative to the working directory
196-
of the current process.
197-
198-
If you specify a ``scope/name`` and ``path``, then the ``path`` cannot
199-
be absolute. It is considered relative to the root of the CodeQL
200-
pack.
201-
202-
To analyze a database using all queries in the `experimental/Security` folder within the `codeql/cpp-queries` CodeQL pack you can use::
203-
204-
codeql database analyze --format=sarif-latest --output=results <db> \
205-
codeql/cpp-queries:experimental/Security
206-
207-
To run the `RedundantNullCheckParam.ql` query in the `codeql/cpp-queries` CodeQL pack use::
208-
209-
codeql database analyze --format=sarif-latest --output=results <db> \
210-
'codeql/cpp-queries:experimental/Likely Bugs/RedundantNullCheckParam.ql'
211-
212-
To analyze your database using the `cpp-security-and-quality.qls` query suite from a version of the `codeql/cpp-queries` CodeQL pack that is >= 0.0.3 and < 0.1.0 (the highest compatible version will be chosen) you can use::
213-
214-
codeql database analyze --format=sarif-latest --output=results <db> \
215-
'codeql/cpp-queries@~0.0.3:codeql-suites/cpp-security-and-quality.qls'
216-
217-
For more information about CodeQL packs, see :doc:`About CodeQL Packs <about-codeql-packs>`.
218-
219223
Running query suites
220224
~~~~~~~~~~~~~~~~~~~~
221225

222-
To run a query suite over a CodeQL database for a C/C++ codebase,
226+
To run a query suite on a CodeQL database for a C/C++ codebase,
223227
you could use the following command from the directory containing your database::
224228

225-
codeql database analyze <cpp-database> cpp-code-scanning.qls --format=sarifv2.1.0 --output=cpp-results.sarif
229+
codeql database analyze <cpp-database> codeql/cpp-queries:codeql-suites/cpp-code-scanning.qls --format=sarifv2.1.0 --output=cpp-results.sarif --download
226230

227-
The analysis generates a file in the v2.1.0 SARIF format that is supported by all versions of GitHub.
228-
This file can be uploaded to GitHub by executing ``codeql github upload-results`` or the code scanning API.
231+
This command downloads the ``codeql/cpp-queries`` CodeQL query pack, runs the analysis, and generates a file in the v2.1.0 SARIF format that is supported by all versions of GitHub. This file can be uploaded to GitHub by executing ``codeql github upload-results`` or the code scanning API.
229232
For more information, see `Analyzing a CodeQL database <https://docs.github.com/en/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system#analyzing-a-codeql-database>`__
230233
or `Code scanning API <https://docs.github.com/en/rest/reference/code-scanning>`__ in the GitHub documentation.
231234

232235
CodeQL query suites are ``.qls`` files that use directives to select queries to run
233236
based on certain metadata properties. The standard CodeQL packs have metadata that specify
234237
the location of the query suites used by code scanning, so the CodeQL CLI knows where to find these
235238
suite files automatically, and you don't have to specify the full path on the command line.
236-
For more information, see ":ref:`About CodeQL packs <standard-codeql-packs>`."
237-
238-
The standard query suites are stored at the following paths in
239-
the CodeQL repository::
239+
For more information, see ":ref:`Creating CodeQL query suites <creating-codeql-query-suites>`."
240240

241-
ql/<language>/ql/src/codeql-suites/<language>-code-scanning.qls
241+
Query suites in the core CodeQL query packs are all stored in the ``codeql-suites`` directory of the pack. For example, the ``codeql/cpp-queries`` pack contains the following query suites:
242242

243-
and at the following path in the CodeQL for Go repository::
243+
* ``cpp-code-scanning.qls`` - Standard Code Scanning queries for C++.
244+
* ``cpp-security-and-quality`` - Security-and-quality queries for C++.
245+
* ``cpp-security-extended`` - Security-extended queries for C++. This suite contains queries that are less precise than the standard security queries, and may find more false-positives.
246+
* ``cpp-lgtm.qls`` - Standard LGTM queries for C++.
247+
* ``cpp-lgtm-full.qls`` - Standard LGTM queries for C++, including less precise queries.
244248

245-
ql/src/codeql-suites/go-code-scanning.qls
246-
247-
The repository also includes the query suites used by `LGTM.com <https://lgtm.com>`__.
248-
These are stored alongside the query suites for code scanning with names of the form: ``<language>-lgtm.qls``.
249+
You can see the sources for these query suites in the `CodeQL repository <https://github.com/github/codeql/tree/main/cpp/ql/src/codeql-suites>`. Query suites for other languages are similar.
249250

250251
For information about creating custom query suites, see ":doc:`Creating
251252
CodeQL query suites <creating-codeql-query-suites>`."
@@ -260,11 +261,11 @@ If the analysis found fewer results for standard queries than you expected, revi
260261
Integrating a CodeQL pack into a code scanning workflow in GitHub
261262
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
262263

263-
.. include:: ../reusables/beta-note-package-management.rst
264-
265264
You can use CodeQL query packs in your code scanning setup. This allows you to select query packs published by various sources and use them to analyze your code.
266265
For more information, see "`Using CodeQL query packs in the CodeQL action <https://docs.github.com/en/code-security/secure-coding/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-codeql-query-packs/>`_" or "`Downloading and using CodeQL query packs in your CI system <https://docs.github.com/en/code-security/secure-coding/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system#downloading-and-using-codeql-query-packs>`_."
267266

267+
.. _including-query-help-for-custom-codeql-queries-in-sarif-files:
268+
268269
Including query help for custom CodeQL queries in SARIF files
269270
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
270271

0 commit comments

Comments
 (0)