Merge latest develop

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Author: pombredanne

.github/workflows/scancode-release.yml

@@ -235,7 +235,7 @@ jobs:
     strategy:
       fail-fast: true
       matrix:
-        os: [ubuntu-22.04, ubuntu-22.04, macos-11, macos-12]
+        os: [ubuntu-22.04, ubuntu-22.04, macos-12, macos-13]
         pyver: ["3.8", "3.9", "3.10", "3.11", "3.12"]
 
     steps:
@@ -367,7 +367,7 @@ jobs:
     strategy:
       fail-fast: true
       matrix:
-        os: [macos-11, macos-12]
+        os: [macos-12, macos-13]
         pyver: ["3.8", "3.9", "3.10", "3.11", "3.12"]
 
     steps:

CHANGELOG.rst

@@ -34,8 +34,107 @@ v33.0.0 (next next, roadmap)
   of these in other summary plugins.
   See https://github.com/nexB/scancode-toolkit/issues/1745
 
-v32.1.0 (next, roadmap)
-----------------------------
+- Update link references of ownership from nexB to aboutcode-org
+  See https://github.com/aboutcode-org/scancode-toolkit/issues/3885
+
+- New and updated licenses, including support for newly released
+  SPDX license list versions:
+  - SPDX License List 3.25.0:
+    This release of the SPDX license list had 9 new licenses
+    and exceptions, and out of them 5 were present as licenses
+    and 2 were present as rules already. There were 2 new
+    license/exception texts added, and also 1 license was deprecated.
+    For more details see https://github.com/aboutcode-org/scancode-toolkit/pull/3897
+
+- New and improved copyright detection with many false positive removed
+  and refined detection added.
+
+
+v32.2.1 - 2024-07-02
+---------------------
+
+- Add support for parsing resolved packages and dependency relationships
+  from nuget lockfile `packages.lock.json`.
+  See https://github.com/nexB/scancode-toolkit/pull/3825
+
+- Add support for parsing resolved packages and dependency relationships
+  from cocoapods lockfile `Podfile.lock`.
+  See https://github.com/nexB/scancode-toolkit/pull/3827
+
+- Add support for parsing packages and dependency relationships
+  from swift `swift-show-dependencies.deplock` generated by DepLock.
+  See https://github.com/nexB/scancode-toolkit/pull/3829
+
+- Add support for `pip-inspect.deplock` files to parse and store
+  resolved packages and dependency relationships, to statically
+  resolve a python dependency graph.
+  See https://github.com/nexB/scancode.io/issues/1262
+
+- Add support for poetry packages, with poetry specific pyproject.toml
+  support, poetry.lock and package assembly support. Also add support
+  for parsing and storing resolved packages and dependency relationships
+  required to statically resolve poetry dependecy graphs.
+  See https://github.com/nexB/scancode-toolkit/issues/2109
+
+- Add support for pyproject.toml files in python projects.
+  See https://github.com/nexB/scancode-toolkit/issues/3753
+
+- More improved copyright detection, see
+  https://github.com/nexB/scancode-toolkit/pull/3752
+
+- ``scancode-toolkit`` is now installable from the fedora repo.
+  See https://github.com/nexB/scancode-toolkit/pull/3824
+
+v32.2.0 - 2024-06-19
+----------------------
+
+- New and improved package/dependency data:
+  - Added new attribute in DependentPackage `is_direct` to aid
+    package resolution and dependency graph creation.
+  - Added new attributes in PackageData: `is_private` and
+    `is_virtual`. #3102 #3811
+  https://github.com/nexB/scancode-toolkit/pull/3779
+
+- Improved javascript package detection:
+  - Add support for pnpm manifests and lockfiles #3766
+  - Add support for npm, pnpm and yarn workspaces #3746
+  - Improve resolved package and dependencies support in lockfiles for
+    yarn.lock, package-lock.json, and pnpm. #3780
+  - Add support for private packages. #3120
+  - Add support for new dependency scopes across javascript
+  - Lots of misc bugfixes in yarn and npm parsers.
+  https://github.com/nexB/scancode-toolkit/pull/3779
+
+- Improve cargo package detection support with various improvements
+  and bugfixes:
+  - Fix for parser crashing on cargo workspaces
+  - Fix a bug in dependency parsing (we were not returning any dependencies)
+  - Also support getting dependency versions from workspace
+  - Support more attributes from cargo
+  - Better handle workspace data thorugh extra_data attribute
+  See https://github.com/nexB/scancode-toolkit/pull/3783
+
+- We now support parsing the Swift manifest JSON dump and the
+  ``Package.resolved`` file https://github.com/nexB/scancode-toolkit/issues/2657.
+  Run the command below on your local Swift project before running the scan:
+    `swift package dump-package > Package.swift.json && swift package resolve``
+
+- New and updated licenses, including support for newly released
+  SPDX license list versions:
+  - SPDX License List 3.24:
+    This release of the SPDX license list had 25 new licenses
+    and exceptions, and out of them 12 were present as licenses
+    and 5 were present as rules already. There were 3 new
+    license/exception texts added, and the rest 5 were either
+    texts with small variations, additions to texts or several
+    rule texts together. And the rest have been added as new licenses.
+    For more details see https://github.com/nexB/scancode-toolkit/pull/3795
+
+  - More new licenses and rules:
+    - 23 new licenses in https://github.com/nexB/scancode-toolkit/pull/3778
+
+v32.1.0 - 2024-03-23
+---------------------
 
 New CLI options:
 
@@ -71,7 +170,7 @@ Changes in Output Data Structure:
       file-level ``package_data``
     * ``license_detections`` and  ``other_license_detections`` in
       codebase level ``packages``
-  
+
   - On using the CLI option ``--license-text-diagnostics`` there is
     now a new license match attribute ``matched_text_diagnostics``
     with the matched text and highlighted diagnostics, instead of
@@ -80,7 +179,7 @@ Changes in Output Data Structure:
   - A new ``reference_matches`` attribute is added to codebase-level
     ``license_detections`` which is same as the ``matches`` attribute
     in other license detections.
-  
+
   - We now have SPDX license expressions everywhere we have
     ScanCode license expressions for ease of use and adopting
     SPDX everywhere. A new attribute ``license_expression_spdx``
@@ -128,7 +227,7 @@ Changes in Output Data Structure:
   and https://github.com/nexB/scancode-toolkit/issues/3443
   Also improve debian manifests parsing and purl parsing from
   filenames. Support for https://github.com/nexB/purldb/issues/245
-  Bumps debian-inspector to v31.1.0 
+  Bumps debian-inspector to v31.1.0
 
 - Bump commoncode to v31.0.3
 

README.rst

@@ -89,7 +89,7 @@ Why use ScanCode?
   InstallShield installers, iOS apps, ISO images, Apache IVY, JBoss Sar,
   R CRAN, Apache Maven, Meteor, Mozilla extensions, MSI installers,
   JavaScript npm packages, package-lock.json, yarn.lock, NSIS Installers,
-  NugGet, OPam, Cocoapods, Python PyPI setup.py, setup.cfg, and 
+  NuGet, OPam, Cocoapods, Python PyPI setup.py, setup.cfg, and 
   several related lockfile formats, semi structured README
   files such as README.android, README.chromium, README.facebook, README.google,
   README.thirdparty, RPMs, Shell Archives, Squashfs images, Java WAR, Windows
@@ -161,6 +161,8 @@ There are a few common ways to `install ScanCode <https://scancode-toolkit.readt
 - `Run in a Docker container with a git clone and "docker run"
   <https://scancode-toolkit.readthedocs.io/en/latest/getting-started/install.html#installation-via-docker>`_
 
+- In Fedora 40+ you can `dnf install scancode-toolkit`
+
 
 Quick Start
 ===========
@@ -258,4 +260,4 @@ the third-party code used in ScanCode for more details.
 
 .. |release-github-actions| image:: https://github.com/nexB/scancode-toolkit/actions/workflows/scancode-release.yml/badge.svg?event=push
     :target: https://github.com/nexB/scancode-toolkit/actions/workflows/scancode-release.yml
-    :alt: Release tests
+    :alt: Release tests

azure-pipelines.yml

@@ -75,6 +75,33 @@ jobs:
                   tests/licensedcode/test_detection_validate.py \
                   -k TestValidateLicenseExtended5
 
+
+              license_validate_ignorables_1: |
+                venv/bin/pytest -n 3 -vvs --test-suite=validate \
+                  tests/licensedcode/test_detection_validate.py \
+                  -k TestValidateLicenseIgnorableClues1
+
+              license_validate_ignorables_2: |
+                venv/bin/pytest -n 3 -vvs --test-suite=validate \
+                  tests/licensedcode/test_detection_validate.py \
+                  -k TestValidateLicenseIgnorableClues2
+
+              license_validate_ignorables_3: |
+                venv/bin/pytest -n 3 -vvs --test-suite=validate \
+                  tests/licensedcode/test_detection_validate.py \
+                  -k TestValidateLicenseIgnorableClues3
+
+              license_validate_ignorables_4: |
+                venv/bin/pytest -n 3 -vvs --test-suite=validate \
+                  tests/licensedcode/test_detection_validate.py \
+                  -k TestValidateLicenseIgnorableClues4
+
+              license_validate_ignorables_5: |
+                venv/bin/pytest -n 3 -vvs --test-suite=validate \
+                  tests/licensedcode/test_detection_validate.py \
+                  -k TestValidateLicenseIgnorableClues5
+
+
               license_cache: |
                 venv/bin/pytest -n 3 -vvs --test-suite=all \
                   tests/licensedcode/test_zzzz_cache.py --reruns 2
@@ -111,19 +138,18 @@ jobs:
 
     - template: etc/ci/azure-posix.yml
       parameters:
-          job_name: macos11_cpython
-          python_architecture: x64
-          image_name: macOS-11
+          job_name: macos12_cpython
+          image_name: macOS-12
           python_versions: ['3.8', '3.9', '3.10', '3.11', '3.12']
+          python_architecture: x64
           test_suites:
               all: venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py --reruns 2
 
     - template: etc/ci/azure-posix.yml
       parameters:
-          job_name: macos12_cpython
-          image_name: macOS-12
+          job_name: macos13_cpython
+          image_name: macOS-13
           python_versions: ['3.8', '3.9', '3.10', '3.11', '3.12']
-          python_architecture: x64
           test_suites:
               all: venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py --reruns 2
 
@@ -204,19 +230,19 @@ jobs:
           python_versions: ['3.8', '3.9', '3.10', '3.11', '3.12']
           test_suites:
               all: venv/bin/pip install --upgrade-strategy eager --force-reinstall --upgrade -e .[testing] && venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py
-
+  
     - template: etc/ci/azure-posix.yml
       parameters:
-          job_name: macos11_cpython_latest_from_pip
-          image_name: macos-11
+          job_name: macos12_cpython_latest_from_pip
+          image_name: macos-12
           python_versions: ['3.8', '3.9', '3.10', '3.11', '3.12']
           test_suites:
               all: venv/bin/pip install --upgrade-strategy eager --force-reinstall --upgrade -e .[testing] && venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py
 
     - template: etc/ci/azure-posix.yml
       parameters:
-          job_name: macos12_cpython_latest_from_pip
-          image_name: macos-12
+          job_name: macos13_cpython_latest_from_pip
+          image_name: macos-13
           python_versions: ['3.8', '3.9', '3.10', '3.11', '3.12']
           test_suites:
               all: venv/bin/pip install --upgrade-strategy eager --force-reinstall --upgrade -e .[testing] && venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py

docs/source/cli-reference/basic-options.rst

@@ -108,7 +108,7 @@
                             "license_expression": "apache-2.0",
                             "rule_identifier": "apache-2.0_65.RULE",
                             "rule_relevance": 100,
-                            "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/apache-2.0_65.RULE",
+                            "rule_url": "https://github.com/aboutcode-org/scancode-toolkit/tree/develop/src/licensedcode/data/rules/apache-2.0_65.RULE",
                             "matched_text": "License: Apache-2.0"
                         }
                     ],
@@ -587,19 +587,19 @@
 
     A scan example using the ``--license-url-template TEXT`` option ::
 
-        scancode -clpieu --json-pp output.json samples --license-url-template https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/{}.LICENSE
+        scancode -clpieu --json-pp output.json samples --license-url-template https://github.com/aboutcode-org/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/{}.LICENSE
 
     In a normal scan, reference url for "ZLIB License" is as follows::
 
         "reference_url": "https://scancode-licensedb.aboutcode.org/zlib",
 
     After using the option in the following manner::
 
-        ``--license-url-template https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/{}.LICENSE``
+        ``--license-url-template https://github.com/aboutcode-org/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/{}.LICENSE``
 
-    the reference URL changes to this `zlib.LICENSE file <https://github.com/nexB/scancode-toolkit/blob/develop/src/licensedcode/data/licenses/zlib.LICENSE>`_::
+    the reference URL changes to this `zlib.LICENSE file <https://github.com/aboutcode-org/scancode-toolkit/blob/develop/src/licensedcode/data/licenses/zlib.LICENSE>`_::
 
-        "reference_url": "https://github.com/nexB/scancode-toolkit/blob/develop/src/licensedcode/data/licenses/zlib.LICENSE",
+        "reference_url": "https://github.com/aboutcode-org/scancode-toolkit/blob/develop/src/licensedcode/data/licenses/zlib.LICENSE",
 
     The reference URL changes for all detected licenses in the scan, across the scan result file.
 
@@ -691,7 +691,7 @@
                             "license_expression": "unknown-license-reference",
                             "rule_identifier": "lead-in_unknown_30.RULE",
                             "rule_relevance": 100,
-                            "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/lead-in_unknown_30.RULE",
+                            "rule_url": "https://github.com/aboutcode-org/scancode-toolkit/tree/develop/src/licensedcode/data/rules/lead-in_unknown_30.RULE",
                             "matched_text": "dual-licensed under [`
                         },
                         {
@@ -704,7 +704,7 @@
                             "license_expression": "wtfpl-2.0",
                             "rule_identifier": "spdx_license_id_wtfpl_for_wtfpl-2.0.RULE",
                             "rule_relevance": 50,
-                            "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/spdx_license_id_wtfpl_for_wtfpl-2.0.RULE",
+                            "rule_url": "https://github.com/aboutcode-org/scancode-toolkit/tree/develop/src/licensedcode/data/rules/spdx_license_id_wtfpl_for_wtfpl-2.0.RULE",
                             "matched_text": "WTFPL"
                         },
                         {
@@ -717,7 +717,7 @@
                             "license_expression": "wtfpl-2.0",
                             "rule_identifier": "wtfpl-2.0_27.RULE",
                             "rule_relevance": 100,
-                            "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/wtfpl-2.0_27.RULE",
+                            "rule_url": "https://github.com/aboutcode-org/scancode-toolkit/tree/develop/src/licensedcode/data/rules/wtfpl-2.0_27.RULE",
                             "matched_text": "www.wtfpl.net/"
                         },
                         {
@@ -730,7 +730,7 @@
                             "license_expression": "mit",
                             "rule_identifier": "mit_64.RULE",
                             "rule_relevance": 100,
-                            "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/mit_64.RULE",
+                            "rule_url": "https://github.com/aboutcode-org/scancode-toolkit/tree/develop/src/licensedcode/data/rules/mit_64.RULE",
                             "matched_text": "MIT`](https://opensource.org/licenses/MIT)."
                         }
                     ],

docs/source/cli-reference/help-text-options.rst

@@ -79,7 +79,7 @@ The Following Help Text is displayed, i.e. This is the help text for Scancode Ve
       --csv FILE              [DEPRECATED] Write scan output as CSV to FILE. The
                               --csv option is deprecated and will be replaced by new
                               CSV and tabular output formats in the next ScanCode
-                              release. Visit https://github.com/nexB/scancode-
+                              release. Visit https://github.com/aboutcode-org/scancode-
                               toolkit/issues/3043 to provide inputs and feedback.
       --html FILE             Write scan output as HTML to FILE.
       --custom-output FILE    Write scan output to FILE formatted with the custom
@@ -321,7 +321,7 @@ The Following Text is displayed, i.e. This is the available plugins for Scancode
     required_plugins:
     options:
       help_group: output formats, name: csv: --csv
-        help: [DEPRECATED] Write scan output as CSV to FILE. The --csv option is deprecated and will be replaced by new CSV and tabular output formats in the next ScanCode release. Visit https://github.com/nexB/scancode-toolkit/issues/3043 to provide inputs and feedback.
+        help: [DEPRECATED] Write scan output as CSV to FILE. The --csv option is deprecated and will be replaced by new CSV and tabular output formats in the next ScanCode release. Visit https://github.com/aboutcode-org/scancode-toolkit/issues/3043 to provide inputs and feedback.
     doc: None
 
   --------------------------------------------