Split Miden Package

[greenhat]

I want to float the idea of splitting Miden package into multiple package types.

So far, the types of entities that we want the Miden package to hold are:

VM Program:

• Name;
• MastArtifact::Executable(Program)
• PackageManifest (no exports, dependencies only stdlib, VM libraries)

VM Library written in MASM:

• Name;
• MastArtifact::Library(Library)
• PackageManifest (multiple exports, dependencies only stdlib, VM libraries)
• WIT interface file (needed for Rust bindings generation)

Rollup Account:

• Name;
• MastArtifact::Library(Library)
• PackageManifest (multiple exports, multiple dependency digests)
• WIT interface file (needed for Rust bindings generation)
• AccountComponentMetadata (account and storage metadata) for deployment

Rollup Note Script:

• Name;
• MastArtifact::Executable(Program)
• PackageManifest (no exports, multiple dependency digests)
• Note script metadata;

Rollup Tx Script:

• Name;
• MastArtifact::Executable(Program)
• PackageManifest (no exports, multiple dependency digests)
• Tx script metadata;

Although they all share some properties (MAST, exports, dependencies) we want to treat them differently. E.g. only an account can be a dependency to a script, only rollup entities can be deployed, but each has its own deployment strategy, etc. Also, each upholds different invariants: scripts have only one export, program has no exports, only account has storage metadata and WIT interface, etc.

This got me thinking if we would benefit from splitting the Miden package into multiple package types, e.g. program package, library package, rollup (account/script) package.

I'd love to hear your thoughts on this.

As a side note, I'm wondering if we should consider merging PackageManifest exports into Library exports, since they duplicate each other. Besides that, both Program and Library feel "incomplete" since the dependencies (think stdlib, tx kernel) are specified only in PackageManifest. Keep in mind that PackageManifest dependencies are just name and digest. So moving dependencies to the Program and Library might be a good idea.

[bobbinth]

This is something I've been thinking about as well - but not sure yet if it is a good idea or not. A few comments/questions:

• If we go this route, I think in the VM we repo we only need Program and Library and we can move the relevant parts of Package into these structs respectively.
• One question is where the definitions for the other 3 package types would live. Maybe it would be in miden-base? But then the complier would need to depend on miden-base (though, maybe it already does?)

Rollup Note Script:

• Name;
• MastArtifact::Library(Library)
• PackageManifest (1 export, multiple dependency digests)

• Technically, note script can also be MastArtifact::Library(Program) - or at least that's how we think of notes in miden-base.
• We'll have custom metadata for notes too (we just haven't defined it yet). It would be much simpler than for accounts, but we will need to define the "shape" of note inputs, and maybe a couple of other things.

Rollup Tx Script:

• Name;
• MastArtifact::Library(Library)
• PackageManifest (1 export, multiple dependency digests)

• Similarly to note scripts, this can also be MastArtifact::Library(Program).
• We may also need slightly different metadata here - though, it will be even simpler than for notes.

only an account can be a dependency to a script

This is not technically correct. Accounts are in fact AccountComponents - and so, a single account may contain multiple components. This means that a note script can invoke procedures in different account components. We don't have examples of this yet, but this could be fairly common where to execute a note script against some account, the account would need to have more than one component (e.g., basic wallet component + something else).

[greenhat]

This is something I've been thinking about as well - but not sure yet if it is a good idea or not.

I'm not entirely convinced that it is a good idea as well, but I felt it worth considering.

A few comments/questions:

• If we go this route, I think in the VM we repo we only need Program and Library and we can move the relevant parts of Package into these structs respectively.

It makes sense. Exports are already defined in Library and dependencies will make it and Program complete and ready to load into the VM.

• One question is where the definitions for the other 3 package types would live. Maybe it would be in miden-base? But then the complier would need to depend on miden-base (though, maybe it already does?)

Seems fitting, since it's a rollup-specific entities. The compiler will depend on miden-base to get the compiled tx kernel for linking anyway. Besides that, Miden SDK already depends on miden-base for AccountComponentMetadata.

I'm wondering if the account compilation artifact (AccountTemplateLibrary? Or even AccountLibrary?) should be an AccountComponentTemplate + WIT interface file + whatever else we are going to need for compilation and linking.

Rollup Note Script:

• Name;
• MastArtifact::Library(Library)
• PackageManifest (1 export, multiple dependency digests)

• Technically, note script can also be MastArtifact::Library(Program) - or at least that's how we think of notes in miden-base.

Right, I'm still stuck in the mindset that it's a Rust library. I'll update the description.

• We'll have custom metadata for notes too (we just haven't defined it yet). It would be much simpler than for accounts, but we will need to define the "shape" of note inputs, and maybe a couple of other things.

Rollup Tx Script:

• Name;
• MastArtifact::Library(Library)
• PackageManifest (1 export, multiple dependency digests)

• Similarly to note scripts, this can also be MastArtifact::Library(Program).
• We may also need slightly different metadata here - though, it will be even simpler than for notes.

I'll update the description.

only an account can be a dependency to a script

This is not technically correct. Accounts are in fact AccountComponents - and so, a single account may contain multiple components. This means that a note script can invoke procedures in different account components. We don't have examples of this yet, but this could be fairly common where to execute a note script against some account, the account would need to have more than one component (e.g., basic wallet component + something else).

Got it. AccountComponent is a good term, but I'm a bit concerned to using it in the compiler since we already have Wasm components and IR components. What I meant by account as a compiled artifact is an AccountComponentTemplate and notes can have multiples of them as dependencies.
What I wanted to highlight is that note scripts and tx scripts cannot be used as dependencies.

[plafer]

I don't have that much experience playing with the Package and all the different ways it's used as you described, so take my opinion with a sizeable grain of salt.

I liked having a single Package which represents all the possible forms - this is similar to how ELF, windows' .exe, etc work (i.e. an ELF can be in "executable" form, or dynamic library, or whatever else). Also I feel like having a single source of truth of "here are all the variations of what a package can look like" is informative at the very least.

A potential problem too if we start defining different package types in different places is how their serialization format could start diverging. So we'd probably want to use different MAGIC strings to differentiate them. And well I don't know, I prefer them all being a "package" (similar to ELF and others) rather than their own thing albeit similar to one another.

So AFAIU, the workflow you suggest is, assuming we have a serialized package:

1. try to deserialize the package into the type that we care about.
2. If deserialization succeeds, then we're good to go.

Compared to the current workflow:

1. Try to deserialize into a package.
2. Ensure that the package has all the data you need given your use case, is in the right format, etc
3. If that succeeds, we're good to go.

I personally prefer the current option - but then again maybe you've felt the pain of it more than I did.

[greenhat]

@bitwalker What do you think?