Improve the repository format

[ikeycode]

Currently we use a flat index for our repository format, and this works rather well. However, we need to add a couple of layers above this in order to support an "Update forever" approach.

Chiefly, we need to generate a root level index on the repo side which in turn has links to moss format versioned indices.
So, for each versioned release of the repo, the tooling repo side would record the highest used moss format version in all of those packages, and in turn set the minimum required moss version as metadata in the root level index.

Client side we would only look at the highest moss version that we can support, filtering out later moss versions to eliminate any unsupported features / fields creeping into the equation. We will need to decide (soon) how we wish to "cut" releases on the repo side, and establish a shared format in libmoss/vessel/moss.

Lastly, we also need to minimise the time spent refreshing indexes. As we need to support dumb-transport / latent mirror cases, we cannot always rely on http headers for "changed since" usage. It is more likely we need to emit a detached checksum (potentially as a moss format files) that includes a new Checksum type, eliminating the need to hex-encode hashes.

Checksum { type, len, data }

We should also work to incorporate the Checksum type as a replacement for PackageHash meta field in future, by renaming the existing enum member to LegacyPackageHash, accounting for a new PackageChecksum that is strongly typed and robust.

Human explanation:

this is a bit of an explanation re versioned repos.
tlder if we add new things to the format or change it, you'll need a version of moss capable of reading it
so we lockstep each repo version to the moss version
this way we can push out a new moss build that will enable incoming features, and cut a release
then we can land packages using those new features ahead of that cut point
so your client wont see the new packages yet, they'd need to move to the new moss first in the last supported version
it has the effect of staggering the update into chunks
ensuring the right moss is in use

[ikeycode]

When designing for this issue we also need to keep #50 in mind, in terms of being able to sign indices as well as individual stones.

[ermo]

Some more notes:

Ikey Doherty
The problem we have atm is interlinked
We absolutely need versioned repos
Such that we protect the existing 6 volatile users
And migrate from there
With that in place we'll have a bigger safety net
As we can retrofit to D infra
Then even an old iso will install last pkg to hit volatile, and perform staggered updates
Ie volatile will need to be preserved and a new dumping repo for builder network
ermo
So there's a moss side and a services side to this it seems?
Ikey Doherty
Yep, interlinked
But what we will do is a series of clever changes
Such that we'll make a vendor file for volatile that sets an enabled field to false
And it'll override the local added repo in etc from the iso
And that moss update will also drag in a new vendor config for the new version capable repo
Along with version support in that moss
Basically the first "epoch bump"
From thereon we'll be syncing/ releasing explicitly
ermo

Ikey Doherty
Such that we'll make a vendor file for volatile that sets an enabled field to false

So this is for the config what we distribute with ... moss?
Ikey Doherty
Yep
Our configuration files get merged
Without an enabled field in the etc file, it'll be masked
Mildly evil but hey.
We need to pull volatile out of circulation
ermo

Ikey Doherty
Without an enabled field in the etc file, it'll be masked

Well, in this case, this implementation detail works to our advantage. Clever.
(if possibly unintended)
Ikey Doherty
Alternative would be special handling of unversioned repo
Either way, we'll make an upgrade path away from volatile
ermo
I think we should explicitly disable volatile in a visible vendor config FWIW (to make it clear that this is not to be pulled from)
Ikey Doherty
I'd agree with that
And some surrounding comments
Then new iso builds will need to unload their repo definitions after populating
So no more of us abusing /etc
Anyway, I see it going well xD