Project policies #386 (#1440)

Signed-off-by: tdruez <tdruez@nexb.com>

Author: tdruez

CHANGELOG.rst

@@ -13,6 +13,11 @@ v34.9.0 (unreleased)
 - Refactor the policies related code to its own module.
   https://github.com/aboutcode-org/scancode.io/issues/386
 
+- Add support for project-specific license policies and compliance alerts.
+  Enhance Project model to handle policies from local settings, project input
+  "policies.yml" files, or global app settings.
+  https://github.com/aboutcode-org/scancode.io/issues/386
+
 v34.8.3 (2024-10-30)
 --------------------
 

docs/command-line-interface.rst

@@ -285,6 +285,9 @@ your outputs on the host machine when running with Docker.
 .. tip:: To specify a CycloneDX spec version (default to latest), use the syntax
   ``cyclonedx:VERSION`` as format value. For example: ``--format cyclonedx:1.5``.
 
+
+.. _cli_check_compliance:
+
 `$ scanpipe check-compliance --project PROJECT`
 -----------------------------------------------
 

docs/faq.rst

@@ -278,3 +278,8 @@ data older than 7 days::
 
   See :ref:`command_line_interface` chapter for more information about the scanpipe
   command.
+
+How can I provide my license policies ?
+---------------------------------------
+
+For detailed information about the policies system, refer to :ref:`policies`.

docs/index.rst

@@ -44,6 +44,7 @@ you’ll find information on:
     custom-pipelines
     scanpipe-pipes
     project-configuration
+    policies
     data-models
     output-files
     command-line-interface

docs/policies.rst

@@ -0,0 +1,108 @@
+.. _policies:
+
+License Policies and Compliance Alerts
+======================================
+
+ScanCode.io enables you to define **license policies** that check your projects
+against a **compliance system**.
+
+Creating Policies Files
+-----------------------
+
+A valid policies file is required to **enable compliance-related features**.
+
+The policies file, by default named ``policies.yml``, is a **YAML file** with a
+structure similar to the following:
+
+.. code-block:: yaml
+
+    license_policies:
+      - license_key: mit
+        label: Approved License
+        compliance_alert: ''
+      - license_key: mpl-2.0
+        label: Restricted License
+        compliance_alert: warning
+      - license_key: gpl-3.0
+        label: Prohibited License
+        compliance_alert: error
+
+- In the example above, licenses are referenced by the ``license_key``,
+  such as `mit` and `gpl-3.0`, which represent the ScanCode license keys used to
+  match against licenses detected in scan results.
+- Each policy is defined with a ``label`` and a ``compliance_alert``.
+  You can customize the labels as desired.
+- The ``compliance_alert`` field accepts three values:
+
+   - ``''`` (empty string)
+   - ``warning``
+   - ``error``
+
+App Policies
+------------
+
+Policies can be enabled for the entire ScanCode.io app instance or on a per-project
+basis.
+
+By default, ScanCode.io will look for a ``policies.yml`` file in the root of its
+application codebase.
+
+Alternatively, you can specify the location of your policies file in your ``.env`` file
+using the :ref:`scancodeio_settings_policies_file` setting.
+
+If a policies file is found at this location, those policies will be applied to
+all projects in the ScanCode.io instance.
+
+.. tip::
+    Refer to the :ref:`scancodeio_settings` section for a full list of settings,
+    including the policies file setting.
+
+Per-Project Policies
+--------------------
+
+Project-specific policies can be provided via a ``policies.yml`` file as one of the
+project inputs or by defining the ``policies`` value in the
+:ref:`project_configuration`.
+
+Compliance Alerts Ranking
+-------------------------
+
+The compliance system uses a ``Precedence of Policies`` principle, which ensures the
+highest-priority policy is applied in cases where resources or packages have complex
+license expressions:
+
+- **error > warning > missing > '' (empty string)**
+
+This principle means that if a resource has an ``error``, ``warning``, and ``''``
+in its license expression, the overall compliance alert for that resource would be
+``error``.
+
+.. warning::
+    The ``missing`` compliance alert value is applied for licenses not included in the
+    policies file.
+
+Web UI
+------
+
+Compliance alerts are shown directly in the Web user interface in the following
+locations:
+
+* A summary panel in the project detail view:
+
+  .. image:: images/tutorial-policies-compliance-alerts-panel.png
+
+* A dedicated column in the Packages and Resources list tables:
+
+  .. image:: images/tutorial-policies-compliance-alerts-column.png
+
+REST API
+--------
+
+For more details on retrieving compliance data through the REST API, see the
+:ref:`rest_api_compliance` section.
+
+Command Line Interface
+----------------------
+
+A dedicated ``check-compliance`` management command is available. See the
+:ref:`cli_check_compliance` section for more information.

docs/project-configuration.rst

@@ -163,3 +163,15 @@ You can provide ``VCID`` from VulnerableCode or any aliases such as ``CVE`` or
      - OSV-2020-871
      - BIT-django-2024-24680
      - PYSEC-2024-28
+
+.. _project_configuration_settings_policies:
+
+policies
+^^^^^^^^
+
+For detailed information about the policies system, refer to :ref:`policies`.
+
+Instead of providing a separate ``policies.yml`` file, policies can be directly
+defined within the project configuration.
+This can be done through the web UI, see :ref:`user_interface_project_settings`,
+or by using a ``scancode-config.yml`` file.

docs/tutorial_license_policies.rst

@@ -3,111 +3,53 @@
 License Policies and Compliance Alerts
 ======================================
 
-In this tutorial, we'll introduce ScanCode.io's **license policies** and **compliance
-alerts** system and use the **results of a pipeline run** to demonstrate an example
-of the license policies and compliance alerts output.
+In this tutorial, we'll introduce ScanCode.io's **license policies** and
+**compliance alerts** system and use the **results of a pipeline run** to demonstrate
+an example of the license policies and compliance alerts output.
 
-As already mentioned, ScanCode.io automates the process of **Software Composition
-Analysis "SCA"** to identify existing open source components and their license
-compliance data in an application's codebase.
+As already mentioned, ScanCode.io automates the process of
+**Software Composition Analysis "SCA"** to identify existing open source components
+and their license compliance data in an application's codebase.
 
 ScanCode.io also gives users the ability to define a set of **license policies** to
 have their projects checked against with a **compliance system**.
 
-Creating Policies Files
------------------------
+Refer to :ref:`policies` for details about the policies system.
 
-A valid policies file is required to **enable compliance-related features**.
+Instructions
+------------
 
-The policies file, by default ``policies.yml``, is a **YAML file** with a structure
-similar to the following:
+Create a ``policies.yml`` file with the following content:
 
 .. code-block:: yaml
 
     license_policies:
     -   license_key: mit
         label: Approved License
         compliance_alert: ''
-    -   license_key: mpl-2.0
-        label: Restricted License
-        compliance_alert: warning
     -   license_key: gpl-3.0
         label: Prohibited License
         compliance_alert: error
 
-- In the above policies file, licenses are referenced by the ``license_key``,
-  such as mit and gpl-3.0, which represents the ScanCode license key to match
-  against detected licenses in the scan results.
-- A policy is defined with a ``label`` and a ``compliance_alert``.
-  The labels can be customized to your preferred wording.
-- The ``compliance_alert`` accepts 3 values:
-
-   - ``''`` (empty string)
-   - ``warning``
-   - ``error``
-
-Policies File Location
-----------------------
-
-By default, ScanCode.io will look for a ``policies.yml`` file at the root of its
-app codebase.
-
-Alternatively, you can configure the location of policies files using the
-dedicated :ref:`scancodeio_settings_policies_file` setting in your ``.env`` file.
-
-.. tip::
-    Check out our :ref:`scancodeio_settings` section for a comprehensive list of
-    settings including policies file setting.
-
-How Does The Compliance Alert Work?
------------------------------------
-
-The compliance system works by following a ``Precedence of Policies`` principal
-allowing the highest precedence policy to be applied in case of resources or
-packages with complex license expressions:
-
-- **error > warning > missing > '' (empty string)**
-
-This principal means a given resource with ``error AND warning AND ''``
-license expression would have an overall compliance alert of ``error``.
-
-.. warning::
-    The ``missing`` compliance alert value is applied for licenses not present in the
-    policies file.
-
-Example Output
---------------
-
-Create a ``policies.yml`` file in the root directory of your ScanCode.io codebase, with
-the following content:
-
-.. code-block:: yaml
-
-    license_policies:
-    -   license_key: mit
-        label: Approved License
-        compliance_alert: ''
-    -   license_key: gpl-3.0
-        label: Prohibited License
-        compliance_alert: error
-
-Run the following command to create a project and run the ``scan_codebase`` pipeline:
+Run the following command to create a project and run the ``scan_codebase`` pipeline
+(make sure to use the proper path for the policies.yml file):
 
 .. code-block:: bash
 
     $ scanpipe create-project cuckoo-filter-with-policies \
         --input-url https://files.pythonhosted.org/packages/75/fc/f5b2e466d763dcc381d5127b73ffc265e8cdaf39ddafa422b7896e625432/cuckoo_filter-1.0.6.tar.gz \
+        --input-file policies.yml \
         --pipeline scan_codebase \
         --execute
 
 Generate results:
 
 .. code-block:: bash
 
-    $ scanpipe output --project cuckoo-filter-with-policies
+    $ scanpipe output --print --project cuckoo-filter-with-policies
 
 The computed compliance alerts are now included in the results, available for each
-detected licenses, and computed at the codebase resource level, for example:
+detected license, and computed at the codebase resource level, for example:
 
 .. code-block:: json
 
@@ -123,38 +65,46 @@ detected licenses, and computed at the codebase resource level, for example:
             "label": "Recommended License",
             "compliance_alert": ""
           },
-        }
+        },
         {
           "key": "gpl-3.0",
           "name": "GNU General Public License 3.0",
           "policy": {
             "label": "Prohibited License",
             "compliance_alert": "error"
-        },
+          }
+        }
       ],
       "license_expressions": [
-        "mit OR gpl-3.0",
+        "mit OR gpl-3.0"
       ],
       "status": "scanned",
       "name": "README",
       "[...]": "[...]"
     }
 
-Web UI
-------
+Run the ``check-compliance`` command
+------------------------------------
 
-Compliance alerts are visible directly in the Web user interface through the following:
+Run the ``check-compliance`` command to get a listing of the compliance alerts detected
+in the project:
 
-* A summary panel in the project detail view:
-
-.. image:: images/tutorial-policies-compliance-alerts-panel.png
+.. code-block:: bash
 
-* A dedicated column within the Packages and Resources list tables:
+    $ scanpipe check-compliance --project cuckoo-filter-with-policies --verbosity 2
 
-.. image:: images/tutorial-policies-compliance-alerts-column.png
+.. code-block:: bash
 
-REST API
---------
+    4 compliance issues detected on this project.
+    [packages]
+     > ERROR: 3
+       pkg:pypi/cuckoo-filter@.
+       pkg:pypi/cuckoo-filter@1.0.6
+       pkg:pypi/cuckoo-filter@1.0.6
+    [resources]
+     > ERROR: 1
+       cuckoo_filter-1.0.6.tar.gz-extract/cuckoo_filter-1.0.6/README.md
 
-For more details on retrieving compliance data via the REST API, refer to the
-:ref:`rest_api_compliance` section.
+.. tip::
+    In case of compliance alerts, the command returns a non-zero exit code which
+    may be useful to trigger a failure in an automated process.

scanpipe/apps.py

@@ -236,11 +236,6 @@ def set_policies(self):
         else:
             logger.debug(style.WARNING("Policies file not found."))
 
-    @property
-    def policies_enabled(self):
-        """Return True if the policies were provided and loaded properly."""
-        return bool(self.license_policies_index)
-
     def sync_runs_and_jobs(self):
         """Synchronize ``QUEUED`` and ``RUNNING`` Run with their related Jobs."""
         logger.info("Synchronizing QUEUED and RUNNING Run with their related Jobs...")