feature: MiniScript package manager

[JoeStrout]

This is something that could apply to both command-line MiniScript and Mini Micro (as well as possibly other environments, such as Soda). But the focus on this ticket will be on the Mini Micro implementation, since that is where the need is most urgent.

Problem: Managing Libraries and Dependencies

Currently if there are libraries you like to use, you have to manually copy them into each project. This causes friction and is error-prone, especially if you are still working on those libraries, in which case it is far too easy to end up with multiple branches containing different features.

A current work-around is to have a separate library folder and mount that as /usr2, and then include that in env.importPaths. But then when you go to distribute your project, you have to remember to copy the used libraries into your /usr disk, which again is error-prone. And any time you send anyone code, you have to be sure to include the external libraries it uses, or instructions on how the user should get them.

Proposal: MiniScript Package Manager

Let's make a standard package manager (exact name TBD) that makes it easy to define what packages/libraries your project needs, and automatically installs them when you don't already have them. Key features:

• Requirements are version-locked and hash-locked, protecting your project from breaking due to package updates as well as supply-chain attacks where somebody uploads malware in place of a common package.
• Required packages are automatically downloaded and installed when the project is run, so the user doesn't have to worry about it; they can just clone/download and run, and it should Just Work.
• This runtime validation should be quick enough to not significantly impact startup time when the needed packages are already present.
• The system should be secure, not enabling project or package developers to get any greater access to the host system than ordinary MiniScript code already has.

Storage Locations

Phase 1: Local Only
In the initial phase, downloaded packages are always installed in a packages folder at the root level of the project (typically accessed in Mini Micro as /usr/packages), which is auto-created if it doesn't already exist. Note that project authors would typically .gitignore the packages folder, so that used packages are not copied up to GitHub along with their project. But when making a distribution for download, they might choose to include it, so that the distro can run without an internet connection or to save the user download time.

Phase 2: Global Option
As a later feature, we allow the user to define some other directory on the host system where packages may be stored. In Mini Micro, this would probably require some UI (perhaps in the BIOS?) for the user to see/change this path. The requirement system would look first in local packages, then in global; and when installing a previously un-installed package, would default to installing globally (or perhaps this is an option the user can set through a config file in the global directory).

Syntax & Behavior

The core part of this manager would be a require command, which takes a specification for a package that is needed. It would look like this:

require "github.com/JoeStrout/spriteExtras@v0.1.0", "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"

This specifies the library to get via (1) a path to a GitHub (or similar?) project, (2) a release tag, and (3) an archive hash (sha256). This command would appear in some script, before any import that relies on the given package. An alternative would be to give the name of a text file that lists requirements, one per line, e.g.:

[In someScript.ms:]

require "requirements.txt"

[In requirements.txt:]

github.com/JoeStrout/spriteExtras@v0.1.0, e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
github.com/jdoe/doeUtils@v1.0.4, 391567a93439523e8e7b8c38c77554a306ee0513d9dafd84d1a6e190587927f1

In either case, the require command causes the following to happen for each specified package:

1. The local packages folder is checked for a copy of the given package and release. (With Phase 2, the global packages folder is also checked). If found, the found package is re-hashed unless it has a recent hash value stored with it; and then the hash is compared to the given hash.
2. If the package was not found, it is downloaded, unpacked, and hashed.
3. If the hash value does not match, a detailed error is printed, and the program exits.
4. If the package was found or downloaded and the hash matches, then its directory is inserted at the front of the env.importPaths list (and removed from any later position at which it may be found).
5. The return value of the require command is that same directory.

Note that no code is actually imported or run from the package at this time; the code will still need to import or otherwise make use of the downloaded files. An import at this point should just work, thanks to the updated env.importPaths. For other usage — for example, loading images or data files included with the package — the code can look in importPaths for the directory, or get it from the return value of the require command.

Note that it is perfectly fine to require the same package more than once in a run. It should execute very quickly on subsequent calls, just returning the directory where the package can be found and moving that directory to the front of the importPaths list.

Indirect Dependencies

The requirements.txt file may have a second section, after a blank line and "Indirect:" header. Requirements after that point are indirect, i.e., not things the current project is using itself, but things that some package its using requires. These are typically found by looking for a requirements.txt file in the packages you require.

The indirect packages are installed when the requirements.txt file is processed, just like the primary requirements. They are separated only to give the package management tool (see below) more context, i.e., there are commands that rewrite the indirect packages by examining the current direct packages.

Package Helper

An interactive program should be provided to help manage the installed packages. This is a later/nice-to-have feature, as it does nothing the user couldn't do manually, but let's spec it out here anyway. It would have the following functions:

• list: show all installed packages with their versions (and hash?)
• install: manually install a package without a hash code (reporting the resulting hash)
• remove: delete a package from the packages folder(s)
• require: add a package to the requirements.txt file of the current project
• unrequire: remove a package from requirements.txt
• find requirements: searches all .ms files in the current project for require commands, and adds them to the primary requirements in requirements.txt; then rewrites the secondary requirements
• yeet: copy local packages to the global packages folder
• yoink: copy all required packages into the local packages folder

Security Considerations

• The hash of each installed package will be recomputed periodically (how often is configurable via a config file in the packages folder, and defaults to 1 day). This protects against something altering an installed package but leaving the old hash value in the cache.
• When we have a global packages folder, it should be read-only to ordinary MiniScript code. Only the require command and the package helper should be able to write to that directory, and only in very specific ways.

[David-Avila]

Since the requirements.txt file will be read/written using the package manager, it could be read only to the user, avoiding typos and improving security.

And for adding indirect dependencies, another command could be added

[JoeStrout]

I wasn't intending requirements.txt to be mainly written by the package manager. I expect most people will tend to have several libraries they commonly use, and they'll just copy/paste those into the requirements for a new project. And that's actually more secure than using the require package manager tool, since the latter would trust that the current files are correct (unless you explicitly provide a hash, which most people wouldn't do in this mode); whereas a copy/paste from a previous project would ensure that the current files still match the old hash.

[JoeStrout]

Some implementation details...

For a given GitHub owner, repo, and release tag, we can generate a download URL as:

https://github.com/{owner}/{repo}/archive/refs/tags/{tag}.zip

For example:

https://github.com/JoeStrout/miniscript-alphabeta/archive/refs/tags/v1.0.0.zip

This will pull down a zip file, which we'll then have to unpack for use. So, to support this feature, we'll need some new functionality in the host app:

• http get (already in Mini Micro, but would be new to command-line MiniScript)
• unzip
• a fast hash function that can apply to a whole folder hierarchy

[JoeStrout]

On names: after pondering this a while, I think I like hopper (or full name, "Hopper MiniScript Package Management System"). This is both a nod to Grace Hopper, and a reference to gravity-flow containers used in farms and factories for dispensing things in a controlled way. Maybe instead of "packages", the folder containing packages should be called "hopper", and we could refer to the local hopper and the global hopper. Installation would mean putting a package into the hopper, using a package would be dispensing it from the hopper, etc.

[MineRobber9000]

I'd like to suggest allowing users to supply a direct URL to a ZIP file, just because that way people could host their packages anywhere and not be tied to GitHub/GitLab/whatever other "project" URLs we might choose to support. We could still support https://github.com/user/repo@tag (as a shorthand for the aforementioned https://github.com/user/repo/archive/refs/tags/tag.zip), but you could also have https://rando-git-forge.org/~user/repo/some-other-way/to-access-archives/for-a-tag.zip.

As for calculating a hash, my idea (poorly fleshed out as it may be) is to recursively calculate the sha256 sum of each file in the hierarchy and store them in a file like the shasum format (HASH NAME, where the whitespace there is 2 space characters), sort that (by either hash or filename), and then hash it to get a final value. It's probably not the most efficient way to do it, but it should meet the requirements of "if any of these files change the hash will also change", and it's a good starting point to launch off of.

[JoeStrout]

Yeah direct URL to a Zip file is a good idea, and I see no reason not to include support for that.