boulder: binary manifests don't list the files a build will generate?

[ermo]

Inspecting a manifest.jsonc reveals a list of filenames w/full paths that a recipe build will generate.

When I do a moss inspect manifest.x86_64.bin, no such list of files is printed? Why is that?

[tarkah]

We don't add content / layout / index payloads, those are for binary stones.

[ermo]

Just capturing the discussion notes here for reference:

Ikey Doherty
JsonC provides a human relevant diff for us for the files
So we can easily review for weirdness and know whats in it at a glance
Its never held any interest for the build automation as a .bin is just a series of augmented meta payloads, no layout payloads
tarkah
Yeah I just commented on the issue, content / index / layout payloads only go to the binary stone
ermo
if a manifest is a promise, why don't we care what files we promise to create?
consider the case of a source-only index
Ikey Doherty
Because we only care about providers for planning
Those are expressed in meta payload
Ie pkgconfig etc
ermo
you want to know which recipe provides /usr/bin/proprietary-thing
tarkah
Right providers is the guarantee, not the list of files. Things can't depend on an explicit file path in recipe
Ikey Doherty
binary()
Solved problem :)
ermo
ok, /usr/share/man-page
Ikey Doherty
Right now nothing cares for it
If there's no way to depend on it then there's no way to express it
A separate problem is "how do I locate files for a repo?"
Which would be a files index on a binary repo, which isn't a solved issue
As the original vision was melding the indices to augment in both directions
ermo
specifically, a from-source-only repo in this case, i.e. a repo with no associated .stones stored in it (only manifests)
Ikey Doherty
I get what you're saying, you should be able to use whatis/whereis
ermo
but as manifests are supposed to act as "promises" (in some fashion), then why not capture the filenames a manifest will produce? No other metadata than that, mind.
Ikey Doherty
And know what source yields a thing
ermo
yes, exactly.
Ikey Doherty
Problem is we're not Sabayon
The manifest format is deliberately simple and small
Augment recipe with build proof and enough provider information to plan builds
Ie power summit or source builds
Chicken and egg isn't solvable as someone needs to bootstrap first to produce the manifests
That was the compromise
Tldr without something like Bootstrap Bill we're not capable of being purely source based
ie, you can't know what you don't know
ermo
... Bootstrap Bill is that tool you envisioned could rebuild/reboostrap a repo?
(name pulled from Pirates of the Carribean I believe?)
Ikey Doherty
The issue here is one of evolutionary levels
A basic bootstrap requires explicit ordering, easy enough, and name based deps, audited by humans
As they can't know in advance the yield
Our current operating layer is higher, where we have a bootstrapped system that can produce these unpredictable yields
Recording as manifests
ermo
Can we make a distinction on required fields and allowed fields in manifest.bin or is that still too "much"?
meaning manifests may provide filenames yield, but does not require it, for the reasons you just outlined
Ikey Doherty
The issue is they're different payloads
How do you then know which layout belongs to what ?
Without introducing a container type into the format
Even at bootstrap level the files themselves don't exactly matter either
ermo
so it's also an architectural/format issue I suppose.
i.e. it has wider implications.
Ikey Doherty
Its more about chained builds and dep tiers
Step 1 producing a bootstrap is deviation right?
So you produce a temporary toolchain for your target
That's a deviation from the stage3 we ship with
So it's impossible to define in the packaging model
unless you have two separate recipes
(This is achieved in rpm land with conditionals in a single recipe)
Also at some point I'd love if we all agree that packaging libraries makes no sense
And that the repo should be a way to produce built apps.
Not a tapestry exercise ^^
ermo
I think what I'm saying is "how is it that, on a practical level, we can add files just fine to the .jsonc manifest after building, but adding them to the binary manifest after building is not possible, even as optional payloads that only contain the filenames like in .jsonc"?
Ikey Doherty
Because we only write jsonc
Boulder never reads it
It's basically perl
Keep in mind we have no concept of optional payloads when it comes to streaming a binary archive
If it doesn't % 2 then .. do what ?
And the only mechanism we have for recording files in a format legal way is the layout payload
Composed of LayoutEntry in D speak
There's no record tag for a file path to go into meta
And it'd seriously inflate the sizes tbf
ermo
so the sticking point is "the format legal way". And we can't change that yet. OK.
(and whether we want to is also a different discussion)
for the from-source side to be "good", this will probably need to be considered. If we have bigger fish to fry right now, ok.
Ikey Doherty
The tldr is currently we can't record the files as binary
We could, if we had a strong enough use case
Which entails the whole figure out bootstrap thing
ermo
I don't think it's important enough until we are ready to really look at the from-source use case, if push comes to shove.
but it will probably need figuring out at some point within, say, the next year or so (probably >6 months at the very least)
so for now, I'm mentally shelving it FWIW.
as it's more of a nice "what if", than a "we need this yesterday".
Ikey Doherty
We know we can augment and bump