Using PurlDB integration to get package metadata requires lots of manual steps

[rogu-beta]

Exploring workarounds for the issue #121 I have deployed PurlDB and integrated it with DejaCode and ScanCode, but without Matchcode.io and also without ScanCode's PurlDB worker (The latter is deactivated because it attempts to scan every Maven package that was present in PurlDB, that I've been syncing. At over 100.000 packages that is simply not feasible.)

The issue I'm facing is that when I import an SBOM which contains purls, but not the special case of ResolvedUrl, that DejaCode and ScanCode do not attempt to resolve the Purl to a download URL. The consequence is that after the load_sbom pipeline completes, DejaCode fills its inventory, but is unable to start scan_single_package, as it cannot supply the download URL to ScanCode.io

With PurlDB I was hoping that ScanCode.io or DejaCode would automatically retrieve such data when analyzing the SBOM. However, that does not seem to happen. Instead I have to manually add and trigger the populate_purldb pipeline to gather that information. As a whole I need to perform the following steps to get to actually scan the packages:

1. Import the SBOM in DejaCode
   • This will trigger the load_sbom pipeline in ScanCode.io and collect all packages and dependencies
   • DejaCode will populate its inventory based on the results of ScanCode.io
2. Once the DejaCode has finished importing the SBOM a person has to manually trigger the populate_purldb pipeline on the project where the SBOM import happened.
   • PurlDB will try to retrieve the metadata of the package and update its database
   • I'm wondering why this is not happening automatically if ScanCode.io is configured to use PurlDB
   • It seems this is currently not working quite reliably (Exception when running populate_purldb on project where previous run was load_sbom scancode.io#1644)
3. When populate_purldb has completed, we can now trigger Improve Packages from PurlDB in DejaCode
   • This will import the information into the respective package in DejaCode
   • The information from the PurlDB contains the download URL among others
4. Finally all information are present to run Scan All Packages
   • Since the download URL is now present, ScanCode.io can analyze the packages
   • We needs this so that we can identify and later export all licenses texts for the software release

The question:
Are there any configurations that I am missing that would make this possible to work without all the manual steps right from the import of the SBOM? My expectation would be that I can upload the SBOM and have it actually perform the package scans without manual intervention with regular SBOMs. As it stands this does not seem to be possible.

[DennisClark]

@ghsa-retrieval thanks very much fro the detailed analysis, which will help us resolve your problem.