Merge branch 'develop' into v31.2.3-branch-hotfix

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Author: AyanSinhaMahapatra

.gitignore

@@ -95,3 +95,20 @@ tcl
 /00-*.txt
 /z-todo-licenses-*
 
+# Extra ignores from licensedb
+*.pyc
+*.db
+.installed.cfg
+parts
+develop-eggs
+eggs
+downloads
+.settings
+TAGS
+Procfile
+local.cfg
+geckodriver.log
+var
+.metaflow
+selenium
+/dist/

CHANGELOG.rst

@@ -1,19 +1,36 @@
 Changelog
 =========
 
+v33.0.0 (next next, roadmap)
 
+----------------------------
 
-v32.0.0 (next next, roadmap)
-----------------------------------
-
-Package detection:
-~~~~~~~~~~~~~~~~~~
 
 - We now support new package manifest formats:
 
   - OpenWRT packages.
   - Yocto/BitBake .bb recipes.
 
+
+v32.0.0 (next, roadmap)
+-----------------------
+
+Important API changes:
+~~~~~~~~~~~~~~~~~~~~~~
+
+This is a major release with major API and output format changes and signicant
+feature updates.
+
+In particular changed to the output format for the licenses and packages, and
+we changed some of the command line options.
+
+The output format version is now 3.0.0.
+
+
+
+Package detection:
+~~~~~~~~~~~~~~~~~~
+
 - Update ``GemfileLockParser`` to track the gem which the Gemfile.lock is for,
   which we assign to the new ``GemfileLockParser.primary_gem`` field. Update
   ``GemfileLockHandler.parse()`` to handle the case where there is a primary gem
@@ -39,20 +56,135 @@ Package detection:
 
   https://github.com/nexB/scancode-toolkit/issues/3081
 
-License detection:
-~~~~~~~~~~~~~~~~~~~
+- Code for parsing a Maven POM, npm package.json, freebsd manifest and haxelib
+  JSON have been separated into two functions: one that creates a PackageData
+  object from the parsed Resource, and another that calls the previous function
+  and yields the PackageData. This was done such that we can use the package
+  manifest data parsing code outside of the scancode-toolkit context in other
+  libraries.
 
-- There is a major update to license detection where we now combine one or
-  matches in a larger license detecion. This remove a larger number of false
-  positive or ambiguous license detections.
 
+License detection:
+~~~~~~~~~~~~~~~~~~~
 
-- The data structure of the JSON output has changed for licenses. We now
-  return match details once for each matched license expression rather than
-  once for each license in a matched expression. There is a new top-level
-  "license_references" attribute that contains the data details for each
-  detected license only once. This data can contain the reference license text
-  as an option.
+- The SPDX license list has been updated to the latest v3.19
+
+- This is a major update to license detection where we now combine one or more
+  license matches in a larger license detection. This approach improves the
+  accuracy of license detection and removes a larger number of false positive
+  or ambiguous license detections. See for details
+  https://github.com/nexB/scancode-toolkit/issues/2878
+
+- There is a new ``license_detections`` codebase level attribute with all the
+  unique license detections in the whole scan, both in resources and packages.
+  This has the 3 attributes also present in package/resource level license
+  detections: ``license_expression``, ``matches`` and ``detection_log`` and has
+  two additional attributes:
+
+  - ``identifier``: which is the ``license_expression`` with an UUID created out
+    of the detection contents and is the same for same detections.
+
+  - ``count``: Number of times in the codebase this unique license detection
+    was encountered.
+
+- The data structure of the JSON output has changed for licenses at file level:
+
+  - The ``licenses`` attribute is deleted.
+
+  - A new ``for_license_detections`` attribute is aded which references the codebase
+    level unique license detections, and this is a list of ``identifer`` strings from
+    the codebase level license detections it references.
+  
+  - A new ``license_detections`` attribute contains license detections in that file.
+    This object has three attributes: ``license_expression``, ``detection_log``
+    and ``matches``. ``matches`` is a list of license matches and is roughly
+    the same as  ``licenses`` in the previous version with additional structure
+    changes detailed below.
+
+  - A new attribute ``license_clues`` contains license matches with the
+    same data structure as the ``matches`` attribute in ``license_detections``.
+    This contains license matches that are mere clues and where not considered
+    to be a proper conclusive license detection.
+
+  - The ``license_expressions`` list of license expressions is deleted and
+    replaced by a ``detected_license_expression`` single expression.
+    Similarly ``spdx_license_expressions`` was removed and replaced by
+    ``detected_license_expression_spdx``.
+
+  - See `license updates documentation <https://scancode-toolkit.readthedocs.io/en/latest/explanations/license-detection-reference.html#change-in-license-data-format-resource>`_
+    for examples and details.
+
+- The data structure of license attributes in ``package_data`` and the codebase
+  level ``packages`` has been updated accordingly:
+
+  - There is a new ``license_detections`` attribute for the primary, top-level
+    declared licenses of a package and an ``other_license_detections`` attribute
+    for the other secondary detections.
+
+  - The ``license_expression`` is replaced by the ``declared_license_expression``
+    and ``other_license_expression`` attributes with their SPDX counterparts
+    ``declared_license_expression_spdx`` and ``other_license_expression_spdx``.
+    These expressions are parallel to detections.
+
+  - The ``declared_license`` attribute is renamed ``extracted_license_statement``
+    and is now a YAML-encoded string.
+
+    See `license updates documentation <https://scancode-toolkit.readthedocs.io/en/latest/explanations/license-detection-reference.html#change-in-license-data-format-package>`_
+    for examples and details.
+
+- The license matches structure has changed: we used to report one match for each
+  license ``key`` of a matched license expression. We now report instead one
+  single match for each matched license expression, and list the license keys
+  as a ``licenses`` attribute. This avoids data duplication.
+  Inside each match, we list each match and matched rule attributred directly
+  avoiding nesting. See `license updates doc <https://scancode-toolkit.readthedocs.io/en/latest/explanations/license-detection-reference.html#licensematch-result-data>`_
+  for examples and details.
+
+- There are new and codebase level attributes default with `--licenses` to report
+  reference license metadata and texts once for each license matched across the
+  scan; we now have two codebase level attributes: ``license_references`` and
+  ``license_rule_references`` that list unique detected license and license rules.
+  for examples and details. This reference data is also removed from license matches
+  in all levels i.e. from codebase, package and resource level license detections and
+  resource level license clues.
+  See `license updates documentation <https://scancode-toolkit.readthedocs.io/en/latest/explanations/license-detection-reference.html#comparision-before-after-license-references>`_
+
+- We replaced the ``scancode --reindex-licenses`` command line option with a
+  new separate command named ``scancode-reindex-licenses``.
+
+  - The ``--reindex-licenses-for-all-languages`` CLI option is also moved to
+    the ``scancode-reindex-licenses`` command as an option ``--all-languages``.
+
+  - We can now detect licenses using custom license texts and license rules
+    stored in a directory or packaged as a plugin for consistent reuse and deployment.
+  
+  - There is an ``--additional-directory`` option with the ``scancode-reindex-licenses``
+    command to add the licenses from a directory.
+    
+  - There is also a ``--only-builtin`` option to use ony builtin licenses
+    ignoring any additional license plugins.
+
+  - See https://github.com/nexB/scancode-toolkit/issues/480 for more details.
+
+- We combined the licensedata file and text file of each license in a single
+  file with a .LICENSE extension. The .yml data file is now included at the
+  top of each .LICENSE file as "YAML frontmatter". The same applies to license
+  rules and their .RULE and .yml files. This halves the number of data files
+  from about 60,000 to 30,000. Git line history is preserved for the combined
+   text + yml files.
+
+  - See https://github.com/nexB/scancode-toolkit/issues/3049
+
+- There is a new ``--get-license-data`` scancode command line option to export
+  license data in JSON, YAML and HTML, with indexes and a static website for use
+  in the licensedb web site. This becomes the  API way to getr scancode license
+  data.
+
+  See https://github.com/nexB/scancode-toolkit/issues/2738
+
+- The deprecated "--is-license-text" option has been removed.
+  This is now built-in with the --license-text option and --info
+  and exposed with the "percentage_of_license_text" attribute.
 
 
 v31.2.4 - 2023-01-09

Dockerfile

@@ -37,9 +37,10 @@ WORKDIR /scancode-toolkit
 # Copy sources into docker container
 COPY . /scancode-toolkit
 
-# Run scancode once for initial configuration, with 
-# --reindex-licenses to create the base license index
-RUN ./scancode --reindex-licenses
+# Initial configuration using ./configure, scancode-reindex-licenses to build
+# the base license index
+RUN ./configure \
+ && ./venv/bin/scancode-reindex-licenses
 
 # Add scancode to path
 ENV PATH=/scancode-toolkit:$PATH

azure-pipelines.yml

@@ -33,6 +33,7 @@ jobs:
                   --ignore=tests/licensedcode/test_detection_datadriven2.py \
                   --ignore=tests/licensedcode/test_detection_datadriven3.py \
                   --ignore=tests/licensedcode/test_detection_datadriven4.py \
+                  --ignore=tests/licensedcode/test_additional_license.py \
                   tests/licensedcode
 
               license_datadriven1_2: |
@@ -78,6 +79,18 @@ jobs:
                 venv/bin/pytest -n 3 -vvs --test-suite=all \
                   tests/licensedcode/test_zzzz_cache.py
 
+              # this test runs in isolation because it modifies the actual
+              # license index with additional licenses provided by a plugin
+              # and we use the special --test-suite=plugins marker for these
+              # tests
+              additional_license_combined: |
+                venv/bin/pip install tests/licensedcode/data/additional_licenses/additional_plugin_1/
+                venv/bin/pip install tests/licensedcode/data/additional_licenses/additional_plugin_2/
+                venv/bin/scancode-reindex-licenses \
+                  --additional-directory tests/licensedcode/data/additional_licenses/additional_dir/
+                venv/bin/pytest -vvs --test-suite=plugins \
+                  tests/licensedcode/test_additional_license.py
+
     - template: etc/ci/azure-posix.yml
       parameters:
           job_name: ubuntu18_cpython
@@ -98,18 +111,9 @@ jobs:
 
     - template: etc/ci/azure-posix.yml
       parameters:
-          job_name: macos1015_cpython_1
-          image_name: macos-10.15
-          python_versions: ['3.8']
-          python_architecture: x64
-          test_suites:
-              all: venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py
-
-    - template: etc/ci/azure-posix.yml
-      parameters:
-          job_name: macos1015_cpython_2
-          image_name: macos-10.15
-          python_versions: ['3.9', '3.10']
+          job_name: ubuntu22_cpython
+          image_name: ubuntu-22.04
+          python_versions: ['3.8', '3.9', '3.10']
           python_architecture: x64
           test_suites:
               all: venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py
@@ -194,27 +198,26 @@ jobs:
 # Tests using a plain pip install to get the latest of all wheels
 ################################################################################
 
-
     - template: etc/ci/azure-posix.yml
       parameters:
-          job_name: ubuntu18_cpython_latest_from_pip
-          image_name: ubuntu-18.04
+          job_name: ubuntu22_cpython_latest_from_pip
+          image_name: ubuntu-22.04
           python_versions: ['3.8', '3.9', '3.10']
           test_suites:
               all: venv/bin/pip install --upgrade-strategy eager --force-reinstall --upgrade -e .[dev] && venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py
 
     - template: etc/ci/azure-posix.yml
       parameters:
-          job_name: ubuntu20_cpython_latest_from_pip
-          image_name: ubuntu-20.04
+          job_name: ubuntu18_cpython_latest_from_pip
+          image_name: ubuntu-18.04
           python_versions: ['3.8', '3.9', '3.10']
           test_suites:
               all: venv/bin/pip install --upgrade-strategy eager --force-reinstall --upgrade -e .[dev] && venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py
 
     - template: etc/ci/azure-posix.yml
       parameters:
-          job_name: macos1015_cpython_latest_from_pip
-          image_name: macos-10.15
+          job_name: ubuntu20_cpython_latest_from_pip
+          image_name: ubuntu-20.04
           python_versions: ['3.8', '3.9', '3.10']
           test_suites:
               all: venv/bin/pip install --upgrade-strategy eager --force-reinstall --upgrade -e .[dev] && venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py

conftest.py

@@ -38,6 +38,7 @@
 ################################################################################
 SLOW_TEST = 'scanslow'
 VALIDATION_TEST = 'scanvalidate'
+PLUGINS_TEST = 'scanplugins'
 
 
 def pytest_configure(config):
@@ -53,8 +54,14 @@ def pytest_configure(config):
         ': Mark a ScanCode test as a validation test, super slow, long running test.',
     )
 
+    config.addinivalue_line(
+        'markers',
+        PLUGINS_TEST +
+        ': Mark a ScanCode test as a special CI test to tests installing additional plugins.',
+    )
+
 
-TEST_SUITES = 'standard', 'all', 'validate'
+TEST_SUITES = ('standard', 'all', 'validate', 'plugins',)
 
 
 def pytest_addoption(parser):
@@ -72,9 +79,11 @@ def pytest_addoption(parser):
         help='Select which test suite to run: '
              '"standard" runs the standard test suite designed to run reasonably fast. '
              '"all" runs "standard" and "slow" (long running) tests. '
-             '"validate" runs all the tests. '
+             '"validate" runs all the tests, except the "plugins" tests. '
+             '"plugins" runs special plugins tests. Needs extra setup, and is used only in the CI. '
              'Use the @pytest.mark.scanslow marker to mark a test as "slow" test. '
              'Use the @pytest.mark.scanvalidate marker to mark a test as a "validate" test.'
+             'Use the @pytest.mark.scanplugins marker to mark a test as a "plugins" test.'
         )
 
 ################################################################################
@@ -87,13 +96,19 @@ def pytest_collection_modifyitems(config, items):
     test_suite = config.getvalue('test_suite')
     run_everything = test_suite == 'validate'
     run_slow_test = test_suite in ('all', 'validate')
+    run_only_plugins = test_suite == 'plugins'
 
     tests_to_run = []
     tests_to_skip = []
 
     for item in items:
         is_validate = bool(item.get_closest_marker(VALIDATION_TEST))
         is_slow = bool(item.get_closest_marker(SLOW_TEST))
+        is_plugins = bool(item.get_closest_marker(PLUGINS_TEST))
+
+        if is_plugins and not run_only_plugins:
+            tests_to_skip.append(item)
+            continue
 
         if is_validate and not run_everything:
             tests_to_skip.append(item)