Add support for pipeline "selected_groups" in the run cli #1306 (#1307)

* Add selected_groups in the API RunSerializer #1306

Signed-off-by: tdruez <tdruez@nexb.com>

* Add support for providing selected_groups in the ``run`` cli #1306

Signed-off-by: tdruez <tdruez@nexb.com>

* Rename the Resolver groups with a no-spaces syntax #1306

Signed-off-by: tdruez <tdruez@nexb.com>

* Reset the FIXTURES_REGEN to False #1306

Signed-off-by: tdruez <tdruez@nexb.com>

* Update CLI end-to-end scanning docs to use groups

Reference: #1302
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Fix test failures

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

---------

Signed-off-by: tdruez <tdruez@nexb.com>
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
Co-authored-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Author: tdruez

CHANGELOG.rst

@@ -15,13 +15,13 @@ v34.6.4 (unreleased)
   related work directories created more than a specified number of days ago.
   https://github.com/nexB/scancode.io/issues/1289
 
-- Update the ``inspect_packages`` pipeline to have an optional ``Static Resolver``
+- Update the ``inspect_packages`` pipeline to have an optional ``StaticResolver``
   group to create resolved packages and dependency relationships from lockfiles
   and manifests having pre-resolved dependencies. Also update this pipeline to
   perform package assembly from multiple manifests and files to create
   discovered packages. Also update the ``resolve_dependencies`` pipeline to have
-  the same ``Static Resolver`` group and mode the dynamic resolution part to a new
-  optional ``Dynamic Resolver`` group.
+  the same ``StaticResolver`` group and mode the dynamic resolution part to a new
+  optional ``DynamicResolver`` group.
   See https://github.com/nexB/scancode.io/pull/1244
 
 - Add a new attribute ``is_direct`` to the DiscoveredDependency model and two new
@@ -34,6 +34,9 @@ v34.6.4 (unreleased)
   and improvements see https://github.com/nexB/scancode-toolkit/releases/tag/v32.2.0
   and https://github.com/nexB/scancode-toolkit/releases/tag/v32.2.1
 
+- Add support for providing pipeline "selected_groups" in the ``run`` entry point.
+  https://github.com/nexB/scancode.io/issues/1306
+
 v34.6.3 (2024-06-21)
 --------------------
 

docs/command-line-interface.rst

@@ -354,16 +354,16 @@ Optional arguments:
 
 .. _cli_run:
 
-`$ run PIPELINE_NAME INPUT_LOCATION`
-------------------------------------
+`$ run PIPELINE_NAME [PIPELINE_NAME ...] input_location`
+--------------------------------------------------------
 
 A ``run`` command is available for executing pipelines and printing the results
 without providing any configuration. This can be useful for running a pipeline to get
 the results without the need to persist the data in the database or access the UI to
 review the results.
 
-.. tip:: You can run multiple pipelines by providing their names, comma-separated,
-  such as `pipeline1,pipeline2`.
+.. tip:: You can run multiple pipelines by providing their names, space-separated,
+  such as `pipeline1 pipeline2`.
 
 Optional arguments:
 
@@ -378,12 +378,16 @@ For example, running the ``inspect_packages`` pipeline on a manifest file:
 
     $ run inspect_packages path/to/package.json > results.json
 
+.. tip:: Use the "pipeline_name:group1,group2" syntax to select steps groups::
+
+    $ run inspect_packages:StaticResolver package.json > results.json
+
 In the following example, running the ``scan_codebase`` followed by the
 ``find_vulnerabilities`` pipelines on a codebase directory:
 
 .. code-block:: bash
 
-    $ run scan_codebase,find_vulnerabilities path/to/codebase/ > results.json
+    $ run scan_codebase find_vulnerabilities path/to/codebase/ > results.json
 
 Using a URL as input is also supported:
 

docs/tutorial_cli_end_to_end_scanning_to_dejacode.rst

@@ -3,9 +3,9 @@
 Analyze Codebase End to End with DepLock and DejaCode (Command Line)
 =====================================================================
 
-The focus of this tutorial is to guide you through scanning a codebase end to end, starting with the
-dependency resolution, through the scanning proper, and finally the upload of the scan in DejaCode,
-using DepLock and ScanCode.io.
+The focus of this tutorial is to guide you through scanning a codebase end to end,
+starting with the dependency resolution, through the scanning proper, and finally
+the upload of the scan in DejaCode, using DepLock and ScanCode.io.
 
 This is designed to run a faster, simple **inspect_packages** ScanCode.io pipeline.
 
@@ -50,7 +50,7 @@ The process for this tutorial is to:
 
 
 .. note::
-    Below instructions have been tested on Linux for now.
+    Below instructions have been tested only on Linux for now.
 
 
 Fetch codebase to scan
@@ -146,14 +146,15 @@ Run ScanCode Package Detection
 -------------------------------
 
 Execute the following command to run the ScanCode scanner
-with **inspect_packages** pipeline on codebase in our current directory.
+with **inspect_packages** pipeline with static resolution
+on the codebase in our current directory.
 
 .. code-block:: shell
 
     docker run --rm \
       -v "$(pwd)":/code \
       ghcr.io/nexb/scancode.io:latest \
-      sh -c "run inspect_packages /code" \
+      sh -c "run inspect_packages:StaticResolver /code" \
       > results.json
 
 

scanpipe/api/serializers.py

@@ -113,6 +113,7 @@ class Meta:
             "pipeline_name",
             "status",
             "description",
+            "selected_groups",
             "project",
             "uuid",
             "created_date",

scanpipe/management/commands/run.py

@@ -35,7 +35,16 @@ class Command(BaseCommand):
 
     def add_arguments(self, parser):
         super().add_arguments(parser)
-        parser.add_argument("pipelines", help="Pipelines to run, comma-separated.")
+        parser.add_argument(
+            "pipelines",
+            metavar="PIPELINE_NAME",
+            nargs="+",
+            help=(
+                "One or more pipeline to run. "
+                "The pipelines executed based on their given order. "
+                'Groups can be provided using the "pipeline_name:group1,group2" syntax.'
+            ),
+        )
         parser.add_argument(
             "input_location", help="Input location: file, directory, and URL supported."
         )
@@ -49,7 +58,6 @@ def add_arguments(self, parser):
 
     def handle(self, *args, **options):
         pipelines = options["pipelines"]
-        pipelines = [pipeline_name.strip() for pipeline_name in pipelines.split(",")]
         input_location = options["input_location"]
         output_format = options["format"]
         # Generate a random name for the project if not provided

scanpipe/pipelines/inspect_packages.py

@@ -65,7 +65,7 @@ def scan_for_application_packages(self):
             progress_logger=self.log,
         )
 
-    @group("Static Resolver")
+    @group("StaticResolver")
     def resolve_dependencies(self):
         """
         Create packages and dependency relationships from

scanpipe/pipelines/resolve_dependencies.py

@@ -57,7 +57,7 @@ def get_manifest_inputs(self):
         """Locate package manifest files with a supported package resolver."""
         self.manifest_resources = resolve.get_manifest_resources(self.project)
 
-    @group("Static Resolver")
+    @group("StaticResolver")
     def scan_for_application_packages(self):
         """
         Scan and assemble application packages from package manifests
@@ -70,15 +70,15 @@ def scan_for_application_packages(self):
             progress_logger=self.log,
         )
 
-    @group("Static Resolver")
+    @group("StaticResolver")
     def create_packages_and_dependencies(self):
         """
         Create the statically resolved packages and their dependencies
         in the database.
         """
         scancode.process_package_data(self.project, static_resolve=True)
 
-    @group("Dynamic Resolver")
+    @group("DynamicResolver")
     def get_packages_from_manifest(self):
         """
         Resolve package data from lockfiles/requirement files with package
@@ -91,7 +91,7 @@ def get_packages_from_manifest(self):
             model="get_packages_from_manifest",
         )
 
-    @group("Dynamic Resolver")
+    @group("DynamicResolver")
     def create_resolved_packages(self):
         """
         Create the dynamically resolved packages and their dependencies

scanpipe/tests/data/asgiref/asgiref-3.3.0_load_inventory_expected.json

@@ -15,6 +15,7 @@
         {
           "pipeline_name": "load_inventory",
           "status": "not_started",
+          "selected_groups": null,
           "scancodeio_version": "",
           "task_id": null,
           "task_start_date": null,

scanpipe/tests/data/d2d/about_files/expected.json

@@ -21,6 +21,7 @@
         {
           "pipeline_name": "map_deploy_to_develop",
           "status": "not_started",
+          "selected_groups": null,
           "scancodeio_version": "",
           "task_id": null,
           "task_start_date": null,
@@ -71,9 +72,9 @@
       "notice_text": "",
       "source_packages": [],
       "extra_data": {},
+      "package_uid": "pkg:local-files/analysis-90cb6382/90cb6382-431c-4187-be76-d4f1a2199a2f?uuid=fixed-uid-done-for-testing-5642512d1758",
       "is_private": false,
       "is_virtual": false,
-      "package_uid": "pkg:local-files/analysis-90cb6382/90cb6382-431c-4187-be76-d4f1a2199a2f?uuid=fixed-uid-done-for-testing-5642512d1758",
       "datasource_ids": [],
       "datafile_paths": [],
       "file_references": [],
@@ -124,9 +125,9 @@
           "*flume-ng-node-*.jar-extract/org/apache/flume/node/ConfigurationProvider.class"
         ]
       },
+      "package_uid": "pkg:maven/log4j/log4j@1.2.13?uuid=fixed-uid-done-for-testing-5642512d1758",
       "is_private": false,
       "is_virtual": false,
-      "package_uid": "pkg:maven/log4j/log4j@1.2.13?uuid=fixed-uid-done-for-testing-5642512d1758",
       "datasource_ids": [],
       "datafile_paths": [],
       "file_references": [],

scanpipe/tests/data/d2d/flume-ng-node-d2d.json

@@ -21,6 +21,7 @@
         {
           "pipeline_name": "map_deploy_to_develop",
           "status": "not_started",
+          "selected_groups": null,
           "scancodeio_version": "",
           "task_id": null,
           "task_start_date": null,
@@ -71,9 +72,9 @@
       "notice_text": "",
       "source_packages": [],
       "extra_data": {},
+      "package_uid": "pkg:local-files/analysis-b74fe5df/b74fe5df-e965-415e-ba65-f38421a0695d?uuid=fixed-uid-done-for-testing-5642512d1758",
       "is_private": false,
       "is_virtual": false,
-      "package_uid": "pkg:local-files/analysis-b74fe5df/b74fe5df-e965-415e-ba65-f38421a0695d?uuid=fixed-uid-done-for-testing-5642512d1758",
       "datasource_ids": [],
       "datafile_paths": [],
       "file_references": [],