Integration of Clarity compliance mechanism

docs/policies.rst

@@ -38,6 +38,43 @@ structure similar to the following:
    - ``warning``
    - ``error``
 
+Creating Clarity Thresholds Files
+---------------------------------
+
+A valid clarity thresholds file is required to **enable clarity compliance features**.
+
+The clarity thresholds file, by default named ``policies.yml``, is a **YAML file** with a
+structure similar to the following:
+
+.. code-block:: yaml
+
+    license_clarity_thresholds:
+      91: ok
+      80: warning
+      0: error
+
+- In the example above, the keys ``91``, ``80``, and ``0`` are integer threshold values
+  representing **minimum clarity scores**.
+- The values ``error``, ``warning``, and ``ok`` are the **compliance alert levels** that
+  will be triggered if the project's license clarity score meets or exceeds the
+  corresponding threshold.
+- The thresholds must be listed in **strictly descending order**.
+
+How it works:
+
+- If the clarity score is **91 or above**, the alert is **``ok``**.
+- If the clarity score is **80 to 90**, the alert is **``warning``**.
+- If the clarity score is **below 80**, the alert is **``error``**.
+
+You can adjust the threshold values and alert levels to match your organization's
+compliance requirements.
+
+Accepted values for the alert level:
+
+- ``ok``
+- ``warning``
+- ``error``
+
 App Policies
 ------------
 
@@ -99,7 +136,7 @@ REST API
 --------
 
 For more details on retrieving compliance data through the REST API, see the
-:ref:`rest_api_compliance` section.
+:ref:`rest_api_compliance` section and :ref:`rest_api_clarity_compliance` section.
 
 Command Line Interface
 ----------------------

docs/rest-api.rst

@@ -493,6 +493,29 @@ Data:
         }
     }
 
+.. _rest_api_clarity_compliance:
+
+Clarity Compliance
+^^^^^^^^^^^^^^^^^^
+
+This action returns the **clarity compliance alert** for a project.
+
+The clarity compliance alert is a single value (``ok``, ``warning``, or ``error``) that summarizes
+the project's **license clarity status**, based on the thresholds defined in the ``policies.yml`` file.
+
+``GET /api/projects/6461408c-726c-4b70-aa7a-c9cc9d1c9685/clarity_compliance/``
+
+Data:
+    - ``clarity_compliance_alert``: The overall clarity compliance alert for the project.
+
+      Possible values: ``ok``, ``warning``, ``error``.
+
+.. code-block:: json
+
+    {
+        "clarity_compliance_alert": "warning"
+    }
+
 Reset
 ^^^^^
 

docs/tutorial_license_policies.rst

@@ -83,6 +83,51 @@ detected license, and computed at the codebase resource level, for example:
       "[...]": "[...]"
     }
 
+License Clarity Thresholds and Compliance
+-----------------------------------------
+
+ScanCode.io also supports **license clarity thresholds**, allowing you to enforce
+minimum standards for license detection quality in your codebase. This is managed
+through the ``license_clarity_thresholds`` section in your ``policies.yml`` file.
+
+Defining Clarity Thresholds
+---------------------------
+
+Add a ``license_clarity_thresholds`` section to your ``policies.yml`` file, for example:
+
+.. code-block:: yaml
+
+    license_clarity_thresholds:
+      91: ok
+      80: warning
+      0: error
+
+
+Clarity Compliance in Results
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When you run a pipeline with clarity thresholds defined in your ``policies.yml``,
+the computed clarity compliance alert is included in the project's ``extra_data`` field.
+
+For example:
+
+.. code-block:: json
+
+    "extra_data": {
+      "md5": "d23df4a4",
+      "sha1": "3e9b61cc98c",
+      "size": 3095,
+      "sha256": "abacfc8bcee59067",
+      "sha512": "208f6a83c83a4c770b3c0",
+      "filename": "cuckoo_filter-1.0.6.tar.gz",
+      "sha1_git": "3fdb0f82ad59",
+      "clarity_compliance_alert": "error"
+    }
+
+The ``clarity_compliance_alert`` value (e.g., ``"error"``, ``"warning"``, or ``"ok"``)
+is computed automatically based on the thresholds you configured and reflects the
+overall license clarity status of the scanned codebase.
+
 Run the ``check-compliance`` command
 ------------------------------------
 
@@ -95,7 +140,7 @@ in the project:
 
 .. code-block:: bash
 
-    4 compliance issues detected on this project.
+    5 compliance issues detected on this project.
     [packages]
      > ERROR: 3
        pkg:pypi/cuckoo-filter@.
@@ -104,6 +149,8 @@ in the project:
     [resources]
      > ERROR: 1
        cuckoo_filter-1.0.6.tar.gz-extract/cuckoo_filter-1.0.6/README.md
+    [License Clarity Compliance]
+     > Alert Level: error
 
 .. tip::
     In case of compliance alerts, the command returns a non-zero exit code which

scanpipe/api/views.py

@@ -481,6 +481,22 @@ def compliance(self, request, *args, **kwargs):
         compliance_alerts = get_project_compliance_alerts(project, fail_level)
         return Response({"compliance_alerts": compliance_alerts})
 
+    @action(detail=True, methods=["get"])
+    def clarity_compliance(self, request, *args, **kwargs):
+        """
+        Retrieve the clarity compliance alert for a project.
+
+        This endpoint returns the clarity compliance alert stored in the
+        project's extra_data.
+
+        Example:
+          GET /api/projects/{project_id}/clarity_compliance/
+
+        """
+        project = self.get_object()
+        clarity_alert = (project.extra_data or {}).get("clarity_compliance_alert")
+        return Response({"clarity_compliance_alert": clarity_alert})
+
 
 class RunViewSet(mixins.RetrieveModelMixin, viewsets.GenericViewSet):
     """Add actions to the Run viewset."""

scanpipe/management/commands/check-compliance.py

@@ -74,16 +74,33 @@ def check_compliance(self, fail_level):
             len(issues) for model in alerts.values() for issues in model.values()
         )
 
-        if count and self.verbosity > 0:
-            self.stderr.write(f"{count} compliance issues detected.")
+        clarity_alert = self.get_clarity_compliance_alert()
+        has_clarity_issue = clarity_alert not in (None, "ok")
+
+        total_issues = count + (1 if has_clarity_issue else 0)
+
+        if total_issues and self.verbosity > 0:
+            self.stderr.write(f"{total_issues} compliance issues detected.")
             for label, model in alerts.items():
                 self.stderr.write(f"[{label}]")
                 for severity, entries in model.items():
                     self.stderr.write(f" > {severity.upper()}: {len(entries)}")
                     if self.verbosity > 1:
                         self.stderr.write("   " + "\n   ".join(entries))
 
-        return count > 0
+            if has_clarity_issue:
+                self.stderr.write("[License Clarity Compliance]")
+                self.stderr.write(f" > Alert Level: {clarity_alert}")
+
+        return total_issues > 0
+
+    def get_clarity_compliance_alert(self):
+        """
+        Return the clarity compliance alert value for the project,
+        or None if not set.
+        """
+        extra_data = self.project.extra_data or {}
+        return extra_data.get("clarity_compliance_alert")
 
     def check_vulnerabilities(self):
         packages = self.project.discoveredpackages.vulnerable_ordered()

scanpipe/pipes/license_clarity.py

@@ -41,6 +41,7 @@
 
 from pathlib import Path
 
+from django.conf import settings
 from django.core.exceptions import ValidationError
 
 import saneyaml
@@ -163,3 +164,27 @@ def load_clarity_thresholds_from_file(file_path):
         return load_clarity_thresholds_from_yaml(yaml_content)
     except (OSError, UnicodeDecodeError) as e:
         raise ValidationError(f"Error reading file {file_path}: {e}")
+
+
+def get_project_clarity_thresholds(project):
+    """
+    Get clarity thresholds for a project, checking multiple sources.
+
+    Returns:
+        ClarityThresholdsPolicy or None: Policy object if thresholds are configured
+
+    """
+    if hasattr(project, "get_input_policies_file"):
+        policies_file = project.get_input_policies_file()
+        if policies_file:
+            policy = load_clarity_thresholds_from_file(policies_file)
+            if policy:
+                return policy
+
+    global_policies_file = getattr(settings, "SCANCODEIO_POLICIES_FILE", None)
+    if global_policies_file:
+        policy = load_clarity_thresholds_from_file(global_policies_file)
+        if policy:
+            return policy
+
+    return None

scanpipe/pipes/scancode.py

@@ -52,6 +52,7 @@
 from scanpipe.models import DiscoveredDependency
 from scanpipe.models import DiscoveredPackage
 from scanpipe.pipes import flag
+from scanpipe.pipes.license_clarity import get_project_clarity_thresholds
 
 logger = logging.getLogger("scanpipe.pipes")
 
@@ -931,7 +932,10 @@ def make_results_summary(project, scan_results_location):
     Extract selected sections of the Scan results, such as the `summary`
     `license_clarity_score`, and `license_matches` related data.
     The `key_files` are also collected and injected in the `summary` output.
+    Additionally, store clarity_compliance_alert in project's extra_data.
     """
+    import json
+
     from scanpipe.api.serializers import CodebaseResourceSerializer
     from scanpipe.api.serializers import DiscoveredPackageSerializer
 
@@ -964,4 +968,16 @@ def make_results_summary(project, scan_results_location):
         DiscoveredPackageSerializer(package).data for package in key_files_packages_qs
     ]
 
+    clarity_score = summary.get("license_clarity_score", {}).get("score")
+    if clarity_score is not None:
+        clarity_policy = get_project_clarity_thresholds(project)
+        if clarity_policy:
+            alert = clarity_policy.get_alert_for_score(clarity_score)
+            summary["clarity_compliance_alert"] = alert
+
+            extra_data = project.extra_data or {}
+            extra_data["clarity_compliance_alert"] = alert
+            project.extra_data = extra_data
+            project.save(update_fields=["extra_data"])
+
     return summary

scanpipe/templates/scanpipe/panels/project_compliance.html

@@ -1,11 +1,11 @@
 {% load humanize %}
-{% if compliance_alerts %}
+{% if compliance_alerts or clarity_compliance_alert %}
   <div class="column is-half">
     <nav id="compliance-panel" class="panel is-dark">
       <p class="panel-heading">
         Compliance alerts
       </p>
-      {% for model_name, model_alerts in compliance_alerts.items  %}
+      {% for model_name, model_alerts in compliance_alerts.items %}
         <div class="panel-block">
           <span class="pr-1">
             {{ model_name|title }}
@@ -19,6 +19,20 @@
           {% endfor %}
         </div>
       {% endfor %}
+      {% if clarity_compliance_alert %}
+        <div class="panel-block">
+          <span class="pr-1">
+            License clarity
+          </span>
+          <span class="tag is-rounded ml-1
+            {% if clarity_compliance_alert == 'error' %}is-danger
+            {% elif clarity_compliance_alert == 'warning' %}is-warning
+            {% elif clarity_compliance_alert == 'ok' %}is-success
+            {% else %}is-light{% endif %}">
+            {{ clarity_compliance_alert|title }}
+          </span>
+        </div>
+      {% endif %}
     </nav>
   </div>
-{% endif %}
+{% endif %}

scanpipe/tests/test_commands.py

@@ -1213,6 +1213,48 @@ def test_scanpipe_management_command_check_compliance(self):
         )
         self.assertEqual(expected, out_value)
 
+    def test_scanpipe_management_command_check_clarity_compliance_only(self):
+        project = make_project(name="my_project_clarity")
+
+        project.extra_data = {"clarity_compliance_alert": "error"}
+        project.save(update_fields=["extra_data"])
+
+        out = StringIO()
+        options = ["--project", project.name]
+        with self.assertRaises(SystemExit) as cm:
+            call_command("check-compliance", *options, stderr=out)
+        self.assertEqual(cm.exception.code, 1)
+        out_value = out.getvalue().strip()
+        expected = (
+            "1 compliance issues detected."
+            "\n[License Clarity Compliance]\n > Alert Level: error"
+        )
+        self.assertEqual(expected, out_value)
+
+    def test_scanpipe_management_command_check_both_compliance_and_clarity(self):
+        project = make_project(name="my_project_both")
+
+        make_package(
+            project,
+            package_url="pkg:generic/name@1.0",
+            compliance_alert=CodebaseResource.Compliance.ERROR,
+        )
+        project.extra_data = {"clarity_compliance_alert": "warning"}
+        project.save(update_fields=["extra_data"])
+
+        out = StringIO()
+        options = ["--project", project.name, "--fail-level", "WARNING"]
+        with self.assertRaises(SystemExit) as cm:
+            call_command("check-compliance", *options, stderr=out)
+        self.assertEqual(cm.exception.code, 1)
+        out_value = out.getvalue().strip()
+        expected = (
+            "2 compliance issues detected."
+            "\n[packages]\n > ERROR: 1"
+            "\n[License Clarity Compliance]\n > Alert Level: warning"
+        )
+        self.assertEqual(expected, out_value)
+
     def test_scanpipe_management_command_check_compliance_vulnerabilities(self):
         project = make_project(name="my_project")
         package1 = make_package(project, package_url="pkg:generic/name@1.0")

scanpipe/tests/test_views.py

@@ -997,6 +997,9 @@ def test_scanpipe_views_project_compliance_panel_view(self, mock_policies_enable
             compliance_alert=DiscoveredPackage.Compliance.ERROR,
         )
 
+        self.project1.extra_data = {"clarity_compliance_alert": "warning"}
+        self.project1.save(update_fields=["extra_data"])
+
         mock_policies_enabled.return_value = False
         response = self.client.get(url)
         self.assertEqual(404, response.status_code)
@@ -1005,6 +1008,8 @@ def test_scanpipe_views_project_compliance_panel_view(self, mock_policies_enable
         response = self.client.get(url)
         self.assertContains(response, "Compliance alerts")
         self.assertContains(response, "1 Error")
+        self.assertContains(response, "License clarity")
+        self.assertContains(response, "Warning")
         expected = f"/project/{self.project1.slug}/packages/?compliance_alert=error"
         self.assertContains(response, expected)