Merge latest develop

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Author: pombredanne

.github/workflows/scancode-release.yml

@@ -235,7 +235,7 @@ jobs:
     strategy:
       fail-fast: true
       matrix:
-        os: [ubuntu-22.04, ubuntu-22.04, macos-11, macos-12]
+        os: [ubuntu-22.04, ubuntu-22.04, macos-12, macos-13]
         pyver: ["3.8", "3.9", "3.10", "3.11", "3.12"]
 
     steps:
@@ -367,7 +367,7 @@ jobs:
     strategy:
       fail-fast: true
       matrix:
-        os: [macos-11, macos-12]
+        os: [macos-12, macos-13]
         pyver: ["3.8", "3.9", "3.10", "3.11", "3.12"]
 
     steps:

CHANGELOG.rst

@@ -34,8 +34,103 @@ v33.0.0 (next next, roadmap)
   of these in other summary plugins.
   See https://github.com/nexB/scancode-toolkit/issues/1745
 
-v32.1.0 (next, roadmap)
-----------------------------
+- Update link references of ownership from nexB to aboutcode-org
+  See https://github.com/aboutcode-org/scancode-toolkit/issues/3885
+
+- New and updated licenses, including support for newly released
+  SPDX license list versions:
+  - SPDX License List 3.25.0:
+    This release of the SPDX license list had 9 new licenses
+    and exceptions, and out of them 5 were present as licenses
+    and 2 were present as rules already. There were 2 new
+    license/exception texts added, and also 1 license was deprecated.
+    For more details see https://github.com/aboutcode-org/scancode-toolkit/pull/3897
+
+v32.2.1 - 2024-07-02
+---------------------
+
+- Add support for parsing resolved packages and dependency relationships
+  from nuget lockfile `packages.lock.json`.
+  See https://github.com/nexB/scancode-toolkit/pull/3825
+
+- Add support for parsing resolved packages and dependency relationships
+  from cocoapods lockfile `Podfile.lock`.
+  See https://github.com/nexB/scancode-toolkit/pull/3827
+
+- Add support for parsing packages and dependency relationships
+  from swift `swift-show-dependencies.deplock` generated by DepLock.
+  See https://github.com/nexB/scancode-toolkit/pull/3829
+
+- Add support for `pip-inspect.deplock` files to parse and store
+  resolved packages and dependency relationships, to statically
+  resolve a python dependency graph.
+  See https://github.com/nexB/scancode.io/issues/1262
+
+- Add support for poetry packages, with poetry specific pyproject.toml
+  support, poetry.lock and package assembly support. Also add support
+  for parsing and storing resolved packages and dependency relationships
+  required to statically resolve poetry dependecy graphs.
+  See https://github.com/nexB/scancode-toolkit/issues/2109
+
+- Add support for pyproject.toml files in python projects.
+  See https://github.com/nexB/scancode-toolkit/issues/3753
+
+- More improved copyright detection, see
+  https://github.com/nexB/scancode-toolkit/pull/3752
+
+- ``scancode-toolkit`` is now installable from the fedora repo.
+  See https://github.com/nexB/scancode-toolkit/pull/3824
+
+v32.2.0 - 2024-06-19
+----------------------
+
+- New and improved package/dependency data:
+  - Added new attribute in DependentPackage `is_direct` to aid
+    package resolution and dependency graph creation.
+  - Added new attributes in PackageData: `is_private` and
+    `is_virtual`. #3102 #3811
+  https://github.com/nexB/scancode-toolkit/pull/3779
+
+- Improved javascript package detection:
+  - Add support for pnpm manifests and lockfiles #3766
+  - Add support for npm, pnpm and yarn workspaces #3746
+  - Improve resolved package and dependencies support in lockfiles for
+    yarn.lock, package-lock.json, and pnpm. #3780
+  - Add support for private packages. #3120
+  - Add support for new dependency scopes across javascript
+  - Lots of misc bugfixes in yarn and npm parsers.
+  https://github.com/nexB/scancode-toolkit/pull/3779
+
+- Improve cargo package detection support with various improvements
+  and bugfixes:
+  - Fix for parser crashing on cargo workspaces
+  - Fix a bug in dependency parsing (we were not returning any dependencies)
+  - Also support getting dependency versions from workspace
+  - Support more attributes from cargo
+  - Better handle workspace data thorugh extra_data attribute
+  See https://github.com/nexB/scancode-toolkit/pull/3783
+
+- We now support parsing the Swift manifest JSON dump and the
+  ``Package.resolved`` file https://github.com/nexB/scancode-toolkit/issues/2657.
+  Run the command below on your local Swift project before running the scan:
+    `swift package dump-package > Package.swift.json && swift package resolve``
+
+- New and updated licenses, including support for newly released
+  SPDX license list versions:
+  - SPDX License List 3.24:
+    This release of the SPDX license list had 25 new licenses
+    and exceptions, and out of them 12 were present as licenses
+    and 5 were present as rules already. There were 3 new
+    license/exception texts added, and the rest 5 were either
+    texts with small variations, additions to texts or several
+    rule texts together. And the rest have been added as new licenses.
+    For more details see https://github.com/nexB/scancode-toolkit/pull/3795
+
+  - More new licenses and rules:
+    - 23 new licenses in https://github.com/nexB/scancode-toolkit/pull/3778
+
+v32.1.0 - 2024-03-23
+---------------------
 
 New CLI options:
 
@@ -71,7 +166,7 @@ Changes in Output Data Structure:
       file-level ``package_data``
     * ``license_detections`` and  ``other_license_detections`` in
       codebase level ``packages``
-  
+
   - On using the CLI option ``--license-text-diagnostics`` there is
     now a new license match attribute ``matched_text_diagnostics``
     with the matched text and highlighted diagnostics, instead of
@@ -80,7 +175,7 @@ Changes in Output Data Structure:
   - A new ``reference_matches`` attribute is added to codebase-level
     ``license_detections`` which is same as the ``matches`` attribute
     in other license detections.
-  
+
   - We now have SPDX license expressions everywhere we have
     ScanCode license expressions for ease of use and adopting
     SPDX everywhere. A new attribute ``license_expression_spdx``
@@ -128,7 +223,7 @@ Changes in Output Data Structure:
   and https://github.com/nexB/scancode-toolkit/issues/3443
   Also improve debian manifests parsing and purl parsing from
   filenames. Support for https://github.com/nexB/purldb/issues/245
-  Bumps debian-inspector to v31.1.0 
+  Bumps debian-inspector to v31.1.0
 
 - Bump commoncode to v31.0.3
 

README.rst

@@ -89,7 +89,7 @@ Why use ScanCode?
   InstallShield installers, iOS apps, ISO images, Apache IVY, JBoss Sar,
   R CRAN, Apache Maven, Meteor, Mozilla extensions, MSI installers,
   JavaScript npm packages, package-lock.json, yarn.lock, NSIS Installers,
-  NugGet, OPam, Cocoapods, Python PyPI setup.py, setup.cfg, and 
+  NuGet, OPam, Cocoapods, Python PyPI setup.py, setup.cfg, and 
   several related lockfile formats, semi structured README
   files such as README.android, README.chromium, README.facebook, README.google,
   README.thirdparty, RPMs, Shell Archives, Squashfs images, Java WAR, Windows
@@ -161,6 +161,8 @@ There are a few common ways to `install ScanCode <https://scancode-toolkit.readt
 - `Run in a Docker container with a git clone and "docker run"
   <https://scancode-toolkit.readthedocs.io/en/latest/getting-started/install.html#installation-via-docker>`_
 
+- In Fedora 40+ you can `dnf install scancode-toolkit`
+
 
 Quick Start
 ===========
@@ -258,4 +260,4 @@ the third-party code used in ScanCode for more details.
 
 .. |release-github-actions| image:: https://github.com/nexB/scancode-toolkit/actions/workflows/scancode-release.yml/badge.svg?event=push
     :target: https://github.com/nexB/scancode-toolkit/actions/workflows/scancode-release.yml
-    :alt: Release tests
+    :alt: Release tests

azure-pipelines.yml

@@ -111,19 +111,18 @@ jobs:
 
     - template: etc/ci/azure-posix.yml
       parameters:
-          job_name: macos11_cpython
-          python_architecture: x64
-          image_name: macOS-11
+          job_name: macos12_cpython
+          image_name: macOS-12
           python_versions: ['3.8', '3.9', '3.10', '3.11', '3.12']
+          python_architecture: x64
           test_suites:
               all: venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py --reruns 2
 
     - template: etc/ci/azure-posix.yml
       parameters:
-          job_name: macos12_cpython
-          image_name: macOS-12
+          job_name: macos13_cpython
+          image_name: macOS-13
           python_versions: ['3.8', '3.9', '3.10', '3.11', '3.12']
-          python_architecture: x64
           test_suites:
               all: venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py --reruns 2
 
@@ -204,19 +203,19 @@ jobs:
           python_versions: ['3.8', '3.9', '3.10', '3.11', '3.12']
           test_suites:
               all: venv/bin/pip install --upgrade-strategy eager --force-reinstall --upgrade -e .[testing] && venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py
-
+  
     - template: etc/ci/azure-posix.yml
       parameters:
-          job_name: macos11_cpython_latest_from_pip
-          image_name: macos-11
+          job_name: macos12_cpython_latest_from_pip
+          image_name: macos-12
           python_versions: ['3.8', '3.9', '3.10', '3.11', '3.12']
           test_suites:
               all: venv/bin/pip install --upgrade-strategy eager --force-reinstall --upgrade -e .[testing] && venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py
 
     - template: etc/ci/azure-posix.yml
       parameters:
-          job_name: macos12_cpython_latest_from_pip
-          image_name: macos-12
+          job_name: macos13_cpython_latest_from_pip
+          image_name: macos-13
           python_versions: ['3.8', '3.9', '3.10', '3.11', '3.12']
           test_suites:
               all: venv/bin/pip install --upgrade-strategy eager --force-reinstall --upgrade -e .[testing] && venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py

docs/source/cli-reference/basic-options.rst

@@ -108,7 +108,7 @@
                             "license_expression": "apache-2.0",
                             "rule_identifier": "apache-2.0_65.RULE",
                             "rule_relevance": 100,
-                            "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/apache-2.0_65.RULE",
+                            "rule_url": "https://github.com/aboutcode-org/scancode-toolkit/tree/develop/src/licensedcode/data/rules/apache-2.0_65.RULE",
                             "matched_text": "License: Apache-2.0"
                         }
                     ],
@@ -587,19 +587,19 @@
 
     A scan example using the ``--license-url-template TEXT`` option ::
 
-        scancode -clpieu --json-pp output.json samples --license-url-template https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/{}.LICENSE
+        scancode -clpieu --json-pp output.json samples --license-url-template https://github.com/aboutcode-org/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/{}.LICENSE
 
     In a normal scan, reference url for "ZLIB License" is as follows::
 
         "reference_url": "https://scancode-licensedb.aboutcode.org/zlib",
 
     After using the option in the following manner::
 
-        ``--license-url-template https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/{}.LICENSE``
+        ``--license-url-template https://github.com/aboutcode-org/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/{}.LICENSE``
 
-    the reference URL changes to this `zlib.LICENSE file <https://github.com/nexB/scancode-toolkit/blob/develop/src/licensedcode/data/licenses/zlib.LICENSE>`_::
+    the reference URL changes to this `zlib.LICENSE file <https://github.com/aboutcode-org/scancode-toolkit/blob/develop/src/licensedcode/data/licenses/zlib.LICENSE>`_::
 
-        "reference_url": "https://github.com/nexB/scancode-toolkit/blob/develop/src/licensedcode/data/licenses/zlib.LICENSE",
+        "reference_url": "https://github.com/aboutcode-org/scancode-toolkit/blob/develop/src/licensedcode/data/licenses/zlib.LICENSE",
 
     The reference URL changes for all detected licenses in the scan, across the scan result file.
 
@@ -691,7 +691,7 @@
                             "license_expression": "unknown-license-reference",
                             "rule_identifier": "lead-in_unknown_30.RULE",
                             "rule_relevance": 100,
-                            "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/lead-in_unknown_30.RULE",
+                            "rule_url": "https://github.com/aboutcode-org/scancode-toolkit/tree/develop/src/licensedcode/data/rules/lead-in_unknown_30.RULE",
                             "matched_text": "dual-licensed under [`
                         },
                         {
@@ -704,7 +704,7 @@
                             "license_expression": "wtfpl-2.0",
                             "rule_identifier": "spdx_license_id_wtfpl_for_wtfpl-2.0.RULE",
                             "rule_relevance": 50,
-                            "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/spdx_license_id_wtfpl_for_wtfpl-2.0.RULE",
+                            "rule_url": "https://github.com/aboutcode-org/scancode-toolkit/tree/develop/src/licensedcode/data/rules/spdx_license_id_wtfpl_for_wtfpl-2.0.RULE",
                             "matched_text": "WTFPL"
                         },
                         {
@@ -717,7 +717,7 @@
                             "license_expression": "wtfpl-2.0",
                             "rule_identifier": "wtfpl-2.0_27.RULE",
                             "rule_relevance": 100,
-                            "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/wtfpl-2.0_27.RULE",
+                            "rule_url": "https://github.com/aboutcode-org/scancode-toolkit/tree/develop/src/licensedcode/data/rules/wtfpl-2.0_27.RULE",
                             "matched_text": "www.wtfpl.net/"
                         },
                         {
@@ -730,7 +730,7 @@
                             "license_expression": "mit",
                             "rule_identifier": "mit_64.RULE",
                             "rule_relevance": 100,
-                            "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/mit_64.RULE",
+                            "rule_url": "https://github.com/aboutcode-org/scancode-toolkit/tree/develop/src/licensedcode/data/rules/mit_64.RULE",
                             "matched_text": "MIT`](https://opensource.org/licenses/MIT)."
                         }
                     ],

docs/source/cli-reference/help-text-options.rst

@@ -79,7 +79,7 @@ The Following Help Text is displayed, i.e. This is the help text for Scancode Ve
       --csv FILE              [DEPRECATED] Write scan output as CSV to FILE. The
                               --csv option is deprecated and will be replaced by new
                               CSV and tabular output formats in the next ScanCode
-                              release. Visit https://github.com/nexB/scancode-
+                              release. Visit https://github.com/aboutcode-org/scancode-
                               toolkit/issues/3043 to provide inputs and feedback.
       --html FILE             Write scan output as HTML to FILE.
       --custom-output FILE    Write scan output to FILE formatted with the custom
@@ -321,7 +321,7 @@ The Following Text is displayed, i.e. This is the available plugins for Scancode
     required_plugins:
     options:
       help_group: output formats, name: csv: --csv
-        help: [DEPRECATED] Write scan output as CSV to FILE. The --csv option is deprecated and will be replaced by new CSV and tabular output formats in the next ScanCode release. Visit https://github.com/nexB/scancode-toolkit/issues/3043 to provide inputs and feedback.
+        help: [DEPRECATED] Write scan output as CSV to FILE. The --csv option is deprecated and will be replaced by new CSV and tabular output formats in the next ScanCode release. Visit https://github.com/aboutcode-org/scancode-toolkit/issues/3043 to provide inputs and feedback.
     doc: None
 
   --------------------------------------------

docs/source/cli-reference/output-format.rst

@@ -58,7 +58,7 @@ following options.
                 "--package": true,
                 "--url": true
               },
-              "notice": "Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/nexB/scancode-toolkit/ for support and download.",
+              "notice": "Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/aboutcode-org/scancode-toolkit/ for support and download.",
               "start_timestamp": "2019-10-19T191117.292858",
               "end_timestamp": "2019-10-19T191219.743133",
               "message": null,
@@ -221,7 +221,7 @@ following options.
 
     The whole Output file looks like::
 
-        {"headers":[{"tool_name":"scancode-toolkit","tool_version":"3.1.1","options":{"input":["samples/"],"--copyright":true,"--email":true,"--info":true,"--json-lines":"output.json","--license":true,"--package":true,"--url":true},"notice":"Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/nexB/scancode-toolkit/ for support and download.","start_timestamp":"2019-10-19T210920.143831","end_timestamp":"2019-10-19T211052.048182","message":null,"errors":[],"extra_data":{"files_count":36}}]}
+        {"headers":[{"tool_name":"scancode-toolkit","tool_version":"3.1.1","options":{"input":["samples/"],"--copyright":true,"--email":true,"--info":true,"--json-lines":"output.json","--license":true,"--package":true,"--url":true},"notice":"Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/aboutcode-org/scancode-toolkit/ for support and download.","start_timestamp":"2019-10-19T210920.143831","end_timestamp":"2019-10-19T211052.048182","message":null,"errors":[],"extra_data":{"files_count":36}}]}
         {"files":[{"path":"samples" ... "scan_errors":[]}]}
         {"files":[{"path":"samples/README", ... "scan_errors":[]}]}
         {"files":[{"path":"samples/screenshot.png", ... "scan_errors":[]}]}
@@ -340,7 +340,7 @@ Comparing Different ``json`` Output Formats
         ScanCode should be considered or used as legal advice. Consult an Attorney
         for any legal advice.
         ScanCode is a free software code scanning tool from nexB Inc. and others.
-        Visit https://github.com/nexB/scancode-toolkit/ for support and download.</text>
+        Visit https://github.com/aboutcode-org/scancode-toolkit/ for support and download.</text>
 
 
         # Creation Info
@@ -459,7 +459,7 @@ Comparing Different ``json`` Output Formats
 
         This option is deprecated and will be replaced by new CSV and tabular
         output formats in the next ScanCode release. Visit
-        https://github.com/nexB/scancode-toolkit/issues/3043
+        https://github.com/aboutcode-org/scancode-toolkit/issues/3043
         for details and to provide inputs and feedback.
 
     The following code performs a scan on the samples directory, and publishes the results in