Enhance Alpine package scan results

[aalexanderr]

Alpine packages lack some important info like copyrights or where the source code is located. This info can't be gathered from the packages themselves as its just not there. To get this info we need to:
download aports repo & for each pkg check it out on commit specific to alpine package (via fetchcode)
parse APKBUILD aboutcode-org/scancode-toolkit#2541
download package sources (fetchcode) & amend new info to package's scan results

Discussed a bit with @pombredanne
Most likely @quepop will PR it

The question is- should it be standard behavior when alpine based docker is being scanned or should it be a separate pipeline?

[quepop]

Related: aboutcode-org/scancode-toolkit#2541 (comment)

[quepop]

@pombredanne should I PR it?

[pombredanne]

@quepop please do! that's much easier for review
Of note:

• I am not super comfy (that's an understatement) with sourcing arbitrary shell scripts at quepop/scancode.io@2f59440#diff-a6e238494a11c38a54d401da6211e8e3461898f10e22ff6673a7896ffedbbd3eR43
• I am not sure we want the alpine package getter to always fetch packages. If anything I know several users run in an airgap'ed environment and they surely will not want (and will not be able anyway) to fetch things from the outside world

[quepop]

@pombredanne I don't think there is any other way. In order to extract source variable value (or any other value) we need to run the script (APKBUILD file), because in some of the alpine packages, source variable is built in a for loop (not to mention that sometimes bash string formatting is used).

From pkg:alpine/alpine-keys@2.2-r0 APKBUILD file:

_arch_keys="
        aarch64:alpine-devel@lists.alpinelinux.org-58199dcc.rsa.pub
        armhf:alpine-devel@lists.alpinelinux.org-524d27bb.rsa.pub
 
        x86:alpine-devel@lists.alpinelinux.org-5243ef4b.rsa.pub
        x86,x86_64:alpine-devel@lists.alpinelinux.org-4a6a0840.rsa.pub
        x86_64:alpine-devel@lists.alpinelinux.org-5261cecb.rsa.pub
 
        ppc64le:alpine-devel@lists.alpinelinux.org-58cbb476.rsa.pub
 
        s390x:alpine-devel@lists.alpinelinux.org-58e4f17d.rsa.pub
 
        mips64:alpine-devel@lists.alpinelinux.org-5e69ca50.rsa.pub
        "
 
for _i in $_arch_keys; do
        source="$source ${_i#*:}"
done

[pombredanne]

@quepop good point, but do we need these (e.g. these _arch_keys) ?

[quepop]

@pombredanne We need the source variable to be complete (to be able to download sources and scan them for copyrights). I tried to emphasize (by giving an example) that the only way we get a complete source variable every time is by running the script.

[quepop]

@pombredanne Regarding aboutcode-org/scancode-toolkit#2541 (comment), how should I name these new files (pipe, pipeline)? My idea is /pipes/alpine_helper.py and /pipelines/alpine_complement.py

[tdruez]

@quepop There's already a pipes/alpine.py pipes module in which you can add the new functions.
For the pipeline, what about "alpine_packages.py", unless it's too specific?

[quepop]

@tdruez In the case of multi-image projects we cannot match DiscoveredPackage from the database to the image (and alpine version) they are from - db entry for DiscoveredPackage only include project_id (no image_id or image_index field). Alpine version is only present in project.extra_data.images array. It's a problem because in the aports repository every alpine version is a certain branch and we need to pull them and make checkouts (branch and then commit-id) to be able to extract correct APKBUILDs.

My solution is to add a new database field to the DiscoveredPackage class named image_index which would store an index (project.extra_data.images array) of the image it is from from. Is it acceptable?

[tdruez]

@quepop what about adding the extra_data field on the DiscoveredPackage model, for consistency with Project and CodebaseResource models?
Let me know if that would work for you and I'll make that change upstream.

[pombredanne]

Repasting from quepop/scancode.io@2f59440#r52902214 for reference :
@quepop

I have been thinking more about this for nexB#191 and sourcing arbitrary bash script this way is too much of a security concern.

An alternative could include:

• using a container, jail or some sorts of chroot to try to minimize the risk a little
• implement a simple bash/shell script parser and perform parameter expansion

I think 2. is best and much less involved than having a container depdendency.
Furthermore, there could be other variables of interest in an APKBUILD and we need eventually to parse other shell-based manifests to extract metadata such as PKGBUILD (Arch), ebuild (Gentoo) m4 (Autotools) and a few more.

Therefore I am implementing this that can then be reused here:

• using a simplified Pygments shell lexer derived from https://github.com/pygments/pygments/blob/master/pygments/lexers/shell.py
• adding lightweight parsing to recognize variables declaration possibly based on https://github.com/nexB/pygmars/ ... or just hand crafted as this is rather simple once the harder lexing is done above
• doing parameter expansion with @kojiromike https://github.com/kojiromike/parameter-expansion/ and this PR I pushed yesterday Support substring and replacement #8 kojiromike/parameter-expansion#9 to address some bash'ism used and needed for APKGBUILD handling

[pombredanne]

@quepop what about adding the extra_data field on the DiscoveredPackage model, for consistency with Project and CodebaseResource models?
Let me know if that would work for you and I'll make that change upstream.

👍 ... this is being added too to SCTK FWIW