Refine the Custom pipeline documentation #237

Signed-off-by: Thomas Druez <tdruez@nexb.com>

Author: tdruez

docs/scanpipe-pipelines.rst renamed to docs/built-in-pipelines.rst

@@ -1,7 +1,7 @@
-.. _scanpipe_pipelines:
+.. _built_in_pipelines:
 
-Pipelines
-=========
+Built-in Pipelines
+==================
 
 .. _pipeline_base_class:
 
@@ -25,6 +25,8 @@ Root Filesystem Analysis
 .. autoclass:: scanpipe.pipelines.root_filesystems.RootFS()
     :members:
 
+.. _pipeline_scan_codebase:
+
 Scan Codebase
 -------------
 .. autoclass:: scanpipe.pipelines.scan_codebase.ScanCodebase()

docs/custom-pipelines.rst

@@ -3,22 +3,29 @@
 Custom Pipelines
 ================
 
-A pipeline always inherits from the ``Pipeline`` base class :ref:`pipeline_base_class`
-It define steps using the ``steps`` class method.
+- A pipeline is a **Python class** that lives in a Python module as a ``.py`` **file**.
+- A pipeline class **always inherits** from the ``Pipeline`` base class
+  :ref:`pipeline_base_class`, or from another existing pipeline class, such as the
+  :ref:`built_in_pipelines`.
+- It **defines steps** using the ``steps`` classmethod.
+
+See :ref:`pipelines_concept` for more details.
 
 Pipeline registration
 ---------------------
 
-Built-in pipelines are located in scanpipe/pipelines/ and registered during the
-ScanCode.io installation.
+Built-in pipelines are located in :guilabel:`scanpipe/pipelines/` directory and
+registered during the ScanCode.io installation.
 
-Custom pipelines can be added as python files in the TBD/ directory and will be
-automatically registered at runtime.
+Custom pipelines can be added as Python files ``.py`` in the directories defined in
+the :ref:`scancodeio_settings_pipelines_dirs` setting and will be automatically
+registered at runtime.
 
 Create a Pipeline
 -----------------
 
-Create a new Python file ``my_pipeline.py`` in the  TBD/ directory.
+Create a new Python file ``my_pipeline.py`` in the and make sure the directory is
+registered in the :ref:`scancodeio_settings_pipelines_dirs` setting.
 
 .. code-block:: python
 
@@ -41,7 +48,8 @@ Create a new Python file ``my_pipeline.py`` in the  TBD/ directory.
 
 
 .. tip::
-    Have a look in the scanpipe/pipelines/ directory for more pipeline examples.
+    Have a look in the :guilabel:`scanpipe/pipelines/` directory for more pipeline
+    examples.
 
 Modify existing Pipelines
 -------------------------
@@ -64,7 +72,7 @@ You may want to override existing steps, add new ones, and remove some.
                 cls.run_scancode,
                 cls.build_inventory_from_scan,
 
-                # Commented-out as I'm not interested in a csv output
+                # Commented-out as not interested in a csv output
                 # cls.csv_output,
 
                 # My extra steps
@@ -77,3 +85,64 @@ You may want to override existing steps, add new ones, and remove some.
 
         def extra_step2(self):
             pass
+
+
+Report step example
+-------------------
+
+Example of a custom pipeline based on the built-in :ref:`pipeline_scan_codebase` one
+with an extra reporting step.
+
+Add the following content to a Python file and register its directory in the
+:ref:`scancodeio_settings_pipelines_dirs`.
+
+.. code-block:: python
+
+    from collections import defaultdict
+
+    from jinja2 import Template
+
+    from scanpipe.pipelines.scan_codebase import ScanCodebase
+
+
+    class ScanAndReport(ScanCodebase):
+        """
+        Run the ScanCodebase built-in pipeline steps and generate a licenses report.
+        """
+
+        @classmethod
+        def steps(cls):
+            return ScanCodebase.steps() + (
+                cls.report_licenses_with_resources,
+            )
+
+        # See https://jinja.palletsprojects.com/en/3.0.x/templates/ for documentation
+        report_template = """
+        {% for matched_text, paths in resources.items() -%}
+            {{ matched_text }}
+
+            {% for path in paths -%}
+                {{ path }}
+            {% endfor %}
+
+        {% endfor %}
+        """
+
+        def report_licenses_with_resources(self):
+            """
+            Retrieve codebase resources filtered by license categories,
+            Generate a licenses report file from a template.
+            """
+            categories = ["Commercial", "Copyleft"]
+            resources = self.project.codebaseresources.licenses_categories(categories)
+
+            resources_by_licenses = defaultdict(list)
+            for resource in resources:
+                for license_data in resource.licenses:
+                    matched_text = license_data.get("matched_text")
+                    resources_by_licenses[matched_text].append(resource.path)
+
+            template = Template(self.report_template, lstrip_blocks=True, trim_blocks=True)
+            report_stream = template.stream(resources=resources_by_licenses)
+            report_file = self.project.get_output_file_path("license-report", "txt")
+            report_stream.dump(str(report_file))

docs/index.rst

@@ -31,7 +31,7 @@ you’ll find information on:
     :caption: Reference Documentation
 
     scanpipe-concepts
-    scanpipe-pipelines
+    built-in-pipelines
     custom-pipelines
     scanpipe-pipes
     scanpipe-output

docs/scancodeio-settings.rst

@@ -74,6 +74,8 @@ of parallel processes to 4::
 
     SCANCODE_DEFAULT_OPTIONS=--processes 4,--timeout 120
 
+.. _scancodeio_settings_pipelines_dirs:
+
 SCANCODEIO_PIPELINES_DIRS
 -------------------------
 

docs/scanpipe-concepts.rst

@@ -37,6 +37,7 @@ The following directories exists under this workspace directory:
   scan results, etc.
 - :guilabel:`tmp/` is a scratch pad for temporary files generated during the pipelines runs.
 
+.. _pipelines_concept:
 
 Pipelines
 ---------