Proposal: Support more import assertion types

[GeoffreyBooth]

While working on nodejs/node#40250, the PR that added support for the import assertions syntax in general and required it for JSON imports, @aduh95 and I discussed a follow-up PR that added support for other formats. There are two in particular that we think should be added:

• webassembly: An import of a WebAssembly module would need the assertion just like JSON modules do: import './file.wasm' assert { type: 'webassembly' }. This follows the examples in the import assertions proposal. (WebAssembly imports are currently flagged behind --experimental-wasm-modules and I think this flag should remain, at least until it becomes clear how browsers will handle WebAssembly imports.)

• javascript: Per the HTML spec, an import statement with no type assertion implies assert { type: 'javascript' }. The explicit version should also be supported, though, because there are cases like import(url, { assert: { type: dynamicType } }) where it is useful for Node to accept an explicit assertion type for modules that don’t necessarily need one.

The ES module loader has five formats defined internally (builtin, commonjs, module, json, wasm), each of which is mapped to either an assertion type (currently only 'json', for the json format) or the “implicit type,” which currently means that an import assertion type isn’t allowed for that format. I propose that we map wasm to 'webassembly'; and that we map builtin, commonjs and module to 'javascript'. The 'javascript' type could either be explicitly specified (import './file.js' assert { type: 'javascript' }) or implied (import './file.js').

These changes will align Node with the HTML spec, which is more detailed than the TC39 import assertions proposal. While Node of course doesn’t need to follow both the TC39 and HTML specs, it would be risky to not do so; any ways in which we deviate from the HTML spec could lead to cases where users can’t write code that can run without modification in both platforms. One potential example is if we disallow type: 'webassembly' and browsers require it, or vice versa. At least for the time being, until it becomes clear how import assertions shake out across the ecosystem, it seems prudent for Node’s default behavior to adhere to the most restrictive spec and for us to wait to relax our requirements until it’s clearly safe to do so. (I’m referring to the Node’s default ESM loader; user loaders can modify Node as they wish and violate the spec as they please.)

Another more-strict part of the HTML spec is that the assertion type is part of the module cache key. The TC39 spec proposal allows different ways to handle multiple imports of the same module with different assertions; the HTML spec team settled on this solution, and I think that Node should align with the HTML spec in order to have consistent behavior across platforms. What this means in practice is that assertion types cannot be optional, in the sense that an import without a declared type is treated as if it had included assert { type: 'javascript' }; no other type can be implied. I wrote more about this in a suggestion for the import assertions proposal.

This also positions Node to support new module types in the future. The only other type that I’m aware of any browsers supporting is 'css', which I haven’t read discussion around Node supporting, but currently there is no way to import native (.node) modules in ESM. This could be remedied by following the example of JSON and Wasm, by creating a new loader format and assertion type for such files. cc @MylesBorins

I’m opening this issue so that this thread can be the place for discussion of this proposal, so that we can limit the chatter on the PR that implements this to technical review of the code in that PR; “product” or “feature” debate should please happen here. cc @nodejs/modules @nodejs/loaders

[jkrems]

IIRC in the browser there's also discussions about having two different kinds of WASM imports, one for the Module and one for the Instance?

[GeoffreyBooth]

Do you have any references for this? Or anything pointing to the current thinking for how browsers are likely to support Wasm imports (no assertion type, type: 'webassembly', or type: 'wasm')?

[ljharb]

Assertions are explicitly meant to be optional in the TC39 spec; it would be a mistake for node to decide to adopt constraints that it doesn't have.

[aduh95]

For folks who are not familiar with the context, the reason the HTML spec requires assertions for JSON is because file extension are meaningless for a web browser (they rely on MIME type, which may or may not be in phase with the file extension), and to let users import safely third party JSON modules (e.g. if import data from 'https://cool.api/legitData.json' was allowed, the cool.api server could sneakily serve a JS module instead, and have read/write access to the global objects, etc.).
Node.js doesn't have this limitation (because there are no MIME types for files on the file system – until we add HTTPS imports support), but imho it makes sense to align with browsers like we did when we decided that extensions will be required in the ESM specifiers, and let non-cross-platform behavior for CJS or custom loader hooks.

The reason assertionless WASM Module/Instance may still be on the table is because IIRC the debate hasn't settle whether WASM modules should be able to import JS modules. If WASM module can import JS, having an assertion is not useful because it would have the same "level of privileges" as JS modules. If not, the assertion would be mandatory for the same reasons as JSON modules. (I wasn't part of any of those discussions, if you think I'm not representing the current state of the debate accurately, please correct me).

[bmeck]

@aduh95 data imports are already supported and are affected by MIME.

[GeoffreyBooth]

data imports are already supported and are affected by MIME.

And certainly HTTPS imports will be, of course, and nodejs/node#36328 is in progress. I think once that lands, Node will have many of the same concerns as browsers, and will need the type assertions for the same reasons. I also think it would be very confusing if an assertion is required for some imports but not others based on whether the module is loaded from the filesystem versus over the network.

[ljharb]

I think it would be much more confusing if an application that uses no network loading was forced to pay a tax solely because of the existence of those features.

If they would be, then if anything that’s an argument against adding those features in the first place.

[bmeck]

If they would be, then if anything that’s an argument against adding those features in the first place.

Overall consumer compatibility to run in multiple environments is definitely a bigger concern for me than avoiding having an assertion in my code. If we don't support import assertions for example, we won't be able to write code that runs in both. Node should be working to make code work in a way that aids our users and I don't see explicitly trying to avoid compatibility as a useful thing. Tooling can be run ahead of time in both cases to mitigate w/e UX people want, but having the end result be the one that is maximally compatible is ideal.

[aduh95]

I think it would be much more confusing if an application that uses no network loading was forced to pay a tax solely because of the existence of those features.

If we take the use case of a JS module written as part of an npm package, the author can't know what protocol their code would run from: it could be local FS (from a node_modules folder), or it could be from unpkg.com if/when HTTPS imports will be supported.

[ljharb]

@aduh95 indeed; importers aren't supposed to have to know the format of the module they're importing - requiring an assertion imposes this burden.

[aduh95]

@aduh95 indeed; importers aren't supposed to have to know the format of the module they're importing - requiring an assertion imposes this burden.

That's a valid point, for something like import "some-package", it imposes the package entry point to be of type JavaScript, which is not a restriction when using CJS.
What I meant is if an author writes something like import "./file.json", it would suddenly matter if the module was loaded from the FS or from the network.

[ljharb]

@aduh95 since the specifier explicitly specifies the filesystem, it would only matter if a loader wanted to load it over the network instead, and also wanted to require the assertion.

[weswigham]

Assertions mix an integrity concern into code which is otherwise not concerned with integrity. IMO, assertions really should be fully optional in node (and honestly, reduce application author freedom), and you really should be able to specify them alongside your dependency hashes in whatever app-level manifest you're using to lock down package integrity, rather than inline. The application author should have the freedom to override dependency resolutions without undue burden.

These assertions reduce the freedom application authors have today for what I understand to be no tangible upside in node. So sure, node should understand the syntax and types and be capable of reading it and validating it so it can handle more code intended for a browser, but I don't think it should be encouraged or required - browsers have invented this inline integrity check because they don't have application (or page-level) dependency manifests in use anywhere, and are trying to solve the problem without one. Node already has one in common use, though. I'm really not a fan of this whole "node needs to clone the WHATWG HTML spec behavior and nothing else" direction. If I wanted to use headless chrome, I would.

[weswigham]

What this does bring up though: we may need a package.json entry (maybe as a part of policies?) that suppresses the error from a mismatched assertion type, that way application authors can still, eg, override a js dependency with a more performant native one, even if it changes the package type, as the decision should ultimately be up to the final application author and not the library author.

[bmeck]

While the idea of a module being naively able to be replaced is sound in theory there are problems in practice that require an ESM wrapper for a facade in general practice:

• import and C++ modules don't generally work with named exports since they need to be linked prior to loading is the general idea, and current design of native modules use dlopen()/run with side effects immediately prior to when we would load. It is possible to delay this somewhat but seems like a larger and more concerning undertaking. Even if this is possible, the general contract of the exports coming off the exported target properties would be broken.

• WASM doesn't have the same API signature capabilities as JS and generally must be wrapped to provide the same API as a JS module, but it can statically declare the exported bindings.

So while the underlying implementation of the algorithms provided by modules may be replaced by other more optimized forms, for all the relevant formats currently you still want a JS wrapper around it so the assertion type would be stable.

[GeoffreyBooth]

IMO, assertions really should be fully optional in node

This was discussed in depth in nodejs/node#39921, including a TSC meeting regarding the topic. Basically, making assertions optional introduces significant complexity on Node’s own loading infrastructure regardless of browsers. It was decided by the TSC that import assertions could ship as required for now, with later consideration around making them optional.

These assertions reduce the freedom application authors have today for what I understand to be no tangible upside in node.

I think nodejs/node#40766 (comment) covers well why I don’t think there’s any reduction of freedom involved here. In terms of the tangible upside, it’s clear: Node could unflag JSON and potentially Wasm module support. Since browsers have already shipped support for import of JSON with an assertion, I think we could unflag our own JSON module import support as soon as we judge the new code to be stable. For Wasm, I think we need to wait until the HTML spec settles on how browsers are going to handle it: type: 'webassembly', type: 'wasm', or no type (so implicitly type: 'javascript' per the HTML spec). Us picking one of these options and shipping it, even flagged, is meant to hopefully nudge progress on that front. If they pick one of the other options, we can change our implementation while we’re still flagged. Or if TC39 would like to make a decision and add it to the spec, we can follow that and unflag our implementation with the assurance that browsers will have to follow the same.

What this does bring up though: we may need a package.json entry (maybe as a part of policies?) that suppresses the error from a mismatched assertion type

This is already possible via loaders, and one of the tests in nodejs/node#40250 was a loader that does just that (enabling assertionless import of JSON). I’m not opposed to other solutions such as flags or package.json entries, but that’s more a question of UX than capability.

browsers have invented this inline integrity check because they don’t have application (or page-level) dependency manifests in use anywhere, and are trying to solve the problem without one. Node already has one in common use, though. . . . I’m really not a fan of this whole “node needs to clone the WHATWG HTML spec behavior and nothing else” direction. If I wanted to use headless chrome, I would.

There’s already an open PR for HTTPS imports, and regardless of whether that lands sooner or later I don’t think we want to make design decisions that would preclude the potential support of that feature. Once HTTPS imports are part of Node, we will have many of the same concerns and limitations as browsers; and for compatibility we would generally want to follow the same solutions as them. Or put another way, the least risky approach is to adhere to both the TC39 and HTML specs whenever possible, as then we’ll have confidence that we won’t run into compatibility issues later. Nonstandard behavior could be opted into via loaders, flags, package.json entries or other methods; but Node would be compatible by default.

[ljharb]

Node isn’t constrained by what browsers do - why would we want it to be?

[bmeck]

@ljharb these are not constraints imposed by browsers, there is complexity from purely the node side. Actual initial surface API seems sane, but as @GeoffreyBooth points out in the previous comment, these require more UX investigation due to added implementation complexity and forwards API design spaces.

E.G.

• adding optional / ambiguous types to the assertion means conflict resolution needs to be added to the loader, and multiple conflict resolutions are likely desirable (race vs reject vs cache results)
• we are not trying to be constrainted by browsers and actively are ensuring a user can do things like write a loader to explicitly override the conservative initial behavior of any implementation
• the nature of node isn't purely files, and the behavior differences across things a module's basic functionality like https: vs data: vs file: based locations isn't something I'm seeing as a constraint but a legitimate concern about UX/ability to re-use code/ability to migrate storage location/debugging

I'd suggest all claiming things about this being constraints imposed by browsers please try and see the actual problems that are not being solved by such claims since a variety of these are completely unrelated to anything from web specifications.

Things like package.json don't even do runtime checks so they don't really apply to a runtime check system like import assertions, and generally are done on version ranges so they don't even guarantee what a dependency is. package-lock.json likewise is only at the application root typically, and may not exist. The runtime checks and things for integrity on individual files and cross module linkage isn't common in Node today. So claims about them need to be more clear on how this runtime module to module linkage check would actually be solved by them.

Things like claims that not needing to know the format of the module allowing optimized module routing simply are real in all forwards facing things that I know of for C++, WASM, or even JSON and generally will want/need an ESM wrapper.

Things like claiming there is no upside to preventing code that will fail when it gets served in a way that is compatible with browsers is actively making an assertion that code in node wouldn't be ported to be served in a situation that wants to match (such as loading a module in both a browser and in node). This isn't related to adherence to browsers, but instead is a simple avoidance of a problem for developers.

So, before we claim "things will be possible if we have ambiguous data" we should be clear that this is not something about browsers for most of these concerns.

I do still think and did argue at TC39 to ensure import assertions should be allowed to be optional, but I don't see lots in the comments about the actual implications of doing so except from people on the loaders team currently. I'm becoming convinced I don't want them to be optional based upon various reasons, none of which are from wanting to follow WHATWG.

[weswigham]

Things like package.json don't even do runtime checks so they don't really apply to a runtime check system like import assertions, and generally are done on version ranges so they don't even guarantee what a dependency is. package-lock.json likewise is only at the application root typically, and may not exist.

Precisely - because a runtime check would be wholly extraneous when you control the entire application and can guarantee consistency prior to execution (because you installed and validated everything in advance). So sure, you should have the choice to assert consistency at runtime, but the common scenario in node doesn't require it. Like, if you've already checked that your node_modules sha(s) matches what you expect at installation, why would you do more at runtime? Paranoia?

we are not trying to be constrainted by browsers and actively are ensuring a user can do things like write a loader to explicitly override the conservative initial behavior of any implementation

Sure. But nice defaults go a long way. And it's pretty much a no-go for library code, since "oh, by the way, my llibrary only works with loader X on" is usually a nonstarter. I don't think I've ever seen a ts-node only library - that'd be silly. Loaders, are, in reality, kind of an excuse - nothing that uses loaders will likely ever see broad use - though I imagine that's somewhat the intent.

the nature of node isn't purely files, and the behavior differences across things a module's basic functionality like https: vs data: vs file: based locations isn't something I'm seeing as a constraint but a legitimate concern about UX/ability to re-use code/ability to migrate storage location/debugging

Again, sure, but far and away most code is and will be file based. Data URLs and blobs are esoteric at best, and I, personally, have yet to be convinced that https imports won't be an antipattern in practice. (They'll definitely make security screening for packages much harder!) Given that, I think it's fair to say files are "blessed" - only files can (currently) be the result of node_modules specifier resolutions without something wrapping them, for example, and only the filesystem participates in the traversal for locating those modules.

adding optional / ambiguous types to the assertion means conflict resolution needs to be added to the loader, and multiple conflict resolutions are likely desirable (race vs reject vs cache results)

Technically the loader already has a kind of conflict resolution, since a json document can be loaded by both the cjs and esm loader - any conflict resolution within the esm loader should probably be consistent with that behavior (which is to say: the same underlying instance appears in all possible caches). If implementation complexity was actually a huge concern, I don't think we'd adopt esm at all, tbh - the complexity it has been adding is a massive PITA, for not just node core but the ecosystem as a whole; what's one more cache concern within node on top of that.

Side note: json import assertions are kind of a PITA that get in the way already - they make it impossible to, eg, swap between a dynamic js file on development and a static json document on production in export maps. Definitely need to be able to squelch that error (or, y'know, not have an import force it to be issued in the first place). That there is no way to say "I don't care about the format of this import" is annoying.

Things like claims that not needing to know the format of the module allowing optimized module routing simply are real in all forwards facing things that I know of for C++, WASM, or even JSON and generally will want/need an ESM wrapper.

I mean, wasm modules can provide named members (even if the in and out types are currently only simple numbers, the interface types proposal aims to fix that anyway), and there's no reason a modern c++ node API couldn't allow a native module to present named members directly without an esm wrapper (just need a new API to declare them). BUT! The issue is not the esm wrapper - the issue would be needing to override the native dependency itself - suppose a package actually wraps or uses the native library in some way before reexporting it (as is common, to handle marshalling/free'ing low-level data types) - you'd have to duplicate all that logic into your wrapped override, rather than just overriding the native dependency itself and providing the same API shape - it'd be very non-minimal and quite unfortunate. Ultimately, implementers should be free to make the override at whatever level they feel appropriate, rather than forced by weird technical blocks into a nonminimal decision - implementer freedom should be the north star here, imo, not forced alignment.

[bmeck]

Like, if you've already checked that your node_modules sha(s) matches what you expect at installation, why would you do more at runtime? Paranoia?

Yes, doing this is good because things like module system exploits are fairly common as an attack vector. I don't think "paranoia" is a good way to attempt to attack the argument for using runtime checks.

Sure. But nice defaults go a long way.

There is disagreement on "nice". The goals of the node project technical priorities do include web compatibility and that is part of the disagreement of the trade offs for what is the best thing to do here.

And it's pretty much a no-go for library code, since "oh, by the way, my llibrary only works with loader X on" is usually a nonstarter. I don't think I've ever seen a ts-node only library - that'd be silly. Loaders, are, in reality, kind of an excuse - nothing that uses loaders will likely ever see broad use - though I imagine that's somewhat the intent.

I think you are once again somewhat attacking the efforts of other here and trying to frame it that people's work is silly or not relevant in order to state it has no bearing of concern in this discussion. If the problem is the ability to use code w/o requiring CLI top level declaration of loaders due to developer friction that is much different from coming out and stating like you did that Loaders aren't relevant for broad use.

Again, sure, but far and away most code is and will be file based. Data URLs and blobs are esoteric at best, and I, personally, have yet to be convinced that https imports won't be an antipattern in practice. (They'll definitely make security screening for packages much harder!) Given that, I think it's fair to say files are "blessed" - only files can (currently) be the result of node_modules specifier resolutions without something wrapping them, for example, and only the filesystem participates in the traversal for locating those modules.

Once again, this is trying to state that a potential future is not relevant to be wary of for designing Node. I disagree with this tactic of argumentation as it is simply trying to preclude any future development in directions that have some desire from the ecosystem/community at large. Just because a feature is not going to be used by an individual does not make that feature something that should be allowed to be completely discarded in discussion of potential problems/trade offs being taken here and now.

Technically the loader already has a kind of conflict resolution, since a json document can be loaded by both the cjs and esm loader - any conflict resolution within the esm loader should probably be consistent with that behavior (which is to say: the same underlying instance appears in all possible caches).

I've argued that they shouldn't share the same cache as CJS can invalidate assumptions about JSON files when they do share a cache. So, even this argument that they should share the same cache seems to be in disagreement.

If implementation complexity was actually a huge concern, I don't think we'd adopt esm at all, tbh - the complexity it has been adding is a massive PITA, for not just node core but the ecosystem as a whole; what's one more cache concern within node on top of that.

This seems unrelated to the issue at hand and appears to be an attempt to treat a separate discussion as a sliding slope for allowances in this discussion. It would be good in this case to understand without the claim of ESM being complex how the concerns of import assertions themselves are not a valid concern with added burden.

I mean, wasm modules can provide named members (even if the in and out types are currently only simple numbers, the interface types proposal aims to fix that anyway),

Even with WASM interface types (which are not yet finished or standardized) they are not aiming for 100% coverage of all possible conversions.

and there's no reason a modern c++ node API couldn't allow a native module to present named members directly without an esm wrapper (just need a new API to declare them). BUT! The issue is not the esm wrapper - the issue would be needing to override the native dependency itself - suppose a package actually wraps or uses the native library in some way before reexporting it (as is common, to handle marshalling/free'ing low-level data types) - you'd have to duplicate all that logic into your wrapped override, rather than just overriding the native dependency itself and providing the same API shape - it'd be very non-minimal and quite unfortunate. Ultimately, implementers should be free to make the override at whatever level they feel appropriate, rather than forced by weird technical blocks into a nonminimal decision - implementer freedom should be the north star here, imo, not forced alignment.

I'm unclear on how this is aided by allowing ambiguity in the loaded dependency. It would be good to understand what practical "freedom" this is allowing.

[bmeck]

Side note: json import assertions are kind of a PITA that get in the way already - they make it impossible to, eg, swap between a dynamic js file on development and a static json document on production in export maps. Definitely need to be able to squelch that error (or, y'know, not have an import force it to be issued in the first place). That there is no way to say "I don't care about the format of this import" is annoying.

I'm mildly sympathetic, but it seems a bit like they could just use a JS file in production. So the problem is easily fixed without adding implementation complexity to loading of modules. This wouldn't work though if the dynamic JS file uses named exports for example. So care needs to be taken to stay on that very small safe overlap of default exports.

[weswigham]

Yes, doing this is good because things like module system exploits are fairly common as an attack vector. I don't think "paranoia" is a good way to attempt to attack the argument for using runtime checks.

Again, the point is choice - some developers may be paranoid and want to reiterate integrity checks at runtime, sure (just like how some defer typechecking until runtime), but yet others may be running node on an embedded system where every cycle of overhead costs silicon budget - developers should have the power over what node does wherever possible. Robbing developers of that choice reduces node's flexibility and reduces its suitability for use in certain projects.

There is disagreement on "nice". The goals of the node project technical priorities do include web compatibility and that is part of the disagreement of the trade offs for what is the best thing to do here.

Yep - and web compatibility doesn't imply web-apis-only. (I mean, that's clearly a nonstarter, right?) So long as node understands web-bound code in the same way web runtimes do, allowing more flexibility than what web runtimes offer should be in the wheelhouse of acceptability.

I think you are once again somewhat attacking the efforts of other here and trying to frame it that people's work is silly or not relevant in order to state it has no bearing of concern in this discussion. If the problem is the ability to use code w/o requiring CLI top level declaration of loaders due to developer friction that is much different from coming out and stating like you did that Loaders aren't relevant for broad use.

This is a practical observation of the world as it exists today - cjs loaders exist, and some do see use, but there're not a library-to-library thing, and even if there are plans to change that, I think hinging the designs of other major systems on those plans looks a lot like wishful thinking - minimally, how the ecosystem will react to future "nice esm loaders" is entirely unknown. Like plugins, "You can write a plugin for that" is very often a polite way to say "No, we won't do that", without actually engaging with the underlying design questions, which is why I don't think they're a great thing to point to as a potential solution.

Once again, this is trying to state that a potential future is not relevant to be wary of for designing Node. I disagree with this tactic of argumentation as it is simply trying to preclude any future development in directions that have some desire from the ecosystem/community at large. Just because a feature is not going to be used by an individual does not make that feature something that should be allowed to be completely discarded in discussion of potential problems/trade offs being taken here and now.

It's not an argument about a future, but rather a current reality. Think too far ahead and you just overdesign things, IMO.

I've argued that they shouldn't share the same cache as CJS can invalidate assumptions about JSON files when they do share a cache. So, even this argument that they should share the same cache seems to be in disagreement.

🤷‍♀️ I don't care what exactly they do when multiple loads occur, I'm just saying a choice has already been made, somewhere, so just being consistent with that choice is what matters. The point is just that a behavior is already implied - the implementation within the esm loader simply doesn't exist yet (it only exists across loaders). If you wanna change the behavior then go for it, it just implied intra-loader conflicts should probably be handled the same way.

This seems unrelated to the issue at hand and appears to be an attempt to treat a separate discussion as a sliding slope for allowances in this discussion. It would be good in this case to understand without the claim of ESM being complex how the concerns of import assertions themselves are not a valid concern with added burden.

Oh, it is, it's definitely just a gripe that found it's way in (though you really have to admit the esm implementation in node is complex compared to the cjs implementation - and only having the cjs one (or only the esm one, though that's impractical) would definitely be simpler), but to continue if you enjoy the offtopic content, feel free to expand

I've started to have to field questions on a daily basis about export maps now that we're trying to get people to adopt them, and wow, it is not pretty. I'm definitely feeling like I contributed to a major failure here, since export maps were supposed to be a solution and not a problem in and of themselves, and it's gotten me _really_ on edge about esm as a whole in node. I'm finding that anyone who wasn't involved in developing esm for node has trouble understanding how esm in node works without some detailed and nuanced instruction. I knew it'd be an issue, but I didn't think it'd be _this bad_.

Even with WASM interface types (which are not yet finished or standardized) they are not aiming for 100% coverage of all possible conversions.

Sure, but they cover a lot of them - and a lot of node packages are a single function that takes some bools/numbers/strings in and spits a bool/number/string out - the area covered is quite large. Sure, complex stuff with object references and streams or something may require complex marshalling and wrapping, but the point is that it isn't always required, and that a developer shouldn't need to wrap it (and may not) if they don't need to do any of that.

I'm unclear on how this is aided by allowing ambiguity in the loaded dependency. It would be good to understand what practical "freedom" this is allowing.

If a depends on b depends on c, at the application level, you can decide, at present, if you need to override a behavior ultimately provided by c (and reexported through b and a), at any of the a, b, or c package levels - whatever's easiest and most appropriate. You have full control - you can override a dependency anywhere in the tree and swap something out. With assertions, a library author can assert something about the integrity of c from b (its format) - this prevents an application author from overriding c alone, reducing their choice - now an application author has to override b (to change the assertion), possibly alongside an override of c (to change the behavior). When you consider how deep dependency trees get in the node community, this kind of patching can become untenable rather quickly if the change has to propagate outwards (say, if a also asserted on the type of b, which you changed when overriding it). Essentially a small version of the issue that made TC39 ultimately abandon integrity hash assertions (imagine the fun cycles in the dependency graph cause!), except the integrity hash is essentially only as many bits as there are module types.

I'm mildly sympathetic, but it seems a bit like they could just use a JS file in production.

Imagine, for a moment, that that's not practical because the parser chokes on it because it's too large (meanwhile the json version is fine because the json parser is much more efficient). Or a some other concern, like the static data in prod coming from a file that's written by some legacy system (while the dynamic js file generates dummy data for testing). It's hypotheticals all the way down, but the point is just that node's kinda getting in the way when it doesn't feel like it needs to - you can set this structure up with the tools we already have, but it's very difficult to make the code actually run in both environments (there's probably some way to make it work, but I doubt it'd be as pretty as an import that just gets swapped out).

[GeoffreyBooth]

Re the swapping-in-production theoretical, that’s similar to an example I wrote for the docs in nodejs/node#40785 for when 'javascript' might be useful:

const assertType = process.env.NODE_ENV === 'development' ? 'javascript' : 'json';
const data = await import('dep/data', { assert: { type: assertType } });

[guybedford]

In Chrome, if I try to import with "type": "javascript" I get the following error:

TypeError: "javascript" is not a valid module type.

Is there any implementation bug or implementation intent known here?

"webassembly" as an assertion type should fall under the definition of the Web Assembly ESM integration, which is being tracked in WebAssembly/esm-integration#42. There is currently no clear consensus on whether "webassembly" should or would be an import assertion type per that spec. The concept of equal capability between JS and Wasm implies that it might even not be.

Personally I quite like the concept of Wasm being an opaque implementation detail, and for that reason would tend towards not wanting to see these assertions.

[aduh95]

According to the HTML spec, javascript should be allowed and equivalent to no assertions. My guess is Chrome team doesn't want to implement this precisely because wasm imports assertion hasn't been settled.

Personally I quite like the concept of Wasm being an opaque implementation detail, and for that reason would tend towards not wanting to see these assertions.

I feel the same, and hopefully this is the way the HTML spec will go, too. (not that Node.js has to follow the HTML spec, but if they deem the assertion optional, there would be no reason for us to impose it)

[GeoffreyBooth]

Is there any implementation bug or implementation intent known here?

This has been a topic of discussion for sure. We looked at the Blink source code and it has a “type: 'javascript'“ included along with json and css, but such an import doesn’t work (yet). We’ve been looking for an explanation online and haven’t found one; if you can find out, I’d love to know why Chrome has left this part of the spec unimplemented. My best guess is the same as @aduh95, that they’re waiting to see how Wasm shakes out; or it’s just a bug.

"webassembly" as an assertion type should fall under the definition of the Web Assembly ESM integration, which is being tracked in WebAssembly/esm-integration#42. There is currently no clear consensus on whether "webassembly" should or would be an import assertion type per that spec. The concept of equal capability between JS and Wasm implies that it might even not be.

I’m happy to go with whatever the spec committees decide. We’re behind a flag, so we can easily change this later if they decide to treat Wasm as type: 'wasm' or as the implicit type: 'javascript'. The import assertions proposal contains several examples of type: 'webassembly' so I’m treating that as the definitive source over GitHub issues in other repos, but if others feel that some other source is more definitive I’m happy to adjust. The HTML spec is silent on the issue as far as I can tell. cc @bmeck

The goal here is just to move the ball forward. Currently I feel like we’re stuck in a stalemate where we can’t implement anything until the spec matures but the spec can’t mature until there are test implementations. The proposal on this thread is my best guess at where the TC39 and HTML specs will end up, based on what they say so far and what browsers have shipped. If my guess turns out to be wrong, I’m happy to change; that’s why we have flags. The import assertions proposal is in Stage 3, which I think means that it should be ready for test implementations to shake out what’s missing in the proposal; and I think that’s what we’re doing in this discussion/PR and my suggestion. cc @MylesBorins, one of the champions of the import assertions proposal.

[guybedford]

The WebAssembly assertions all have a caveat pointing to an issue which points to the Web Assembly ESM integration as the source of truth on this, where it remains undecided.

I've posted a follow-up in tc39/proposal-import-attributes#49 to see if we can get guidance on the "javascript" expectation.

@GeoffreyBooth appreciated this work is moving things forward, but when the blockers really are spec questions implementation shouldn't be rushed, and instead spec clarifications should be sought - that is what Stage 3 is about.

[GeoffreyBooth]

when the blockers really are spec questions implementation shouldn’t be rushed, and instead spec clarifications should be sought

I don’t think there’s any clarification needed for type: 'javascript'—it’s already part of the HTML spec, so clearly it’s permitted via the TC39 spec/proposal, and Node might as well support it for compatibility. I would appreciate if TC39 would add it to their spec as well for consistency, but that doesn’t seem like something we need to wait on.

As for type: 'webassembly', I hate to be a downer but I think the likely outcome is that it will be required in browsers, because the conservative approach would be to assume that Wasm might require different permissions than JavaScript, if not now then potentially someday, and therefore Wasm would need an explicit type. Hence why I think we’re better off defaulting to this more-restrictive assumption and relaxing it later if my guess is wrong, rather than the other way around. If the spec eventually settles on the string 'wasm' instead of 'webassembly', we can easily adjust at any time while we’re flagged (and we wouldn’t unflag until it’s settled). This is also probably an HTML spec thing, not a TC39 spec thing, assuming the TC39 spec continues its pattern of allowing hosts to decide.

@guybedford if there’s a way to request clarification for type: 'webassembly', that would be great; do you mind asking, or telling me where to ask?

[guybedford]

I think the "type": "javascript" default assertion treatment in HTML for arbitrary imports will need to change when Web Assembly integrates into the platform anyway, as it will effectively be wrong. This seems like a spec oversight to me.

the conservative approach would be to assume that Wasm might require different permissions than JavaScript

Web Assembly has equal capabilities to JS - the idea that one or the other represents a security difference is simply not true and it's a big leap to assume there is any need from any security perspective for a { type: 'webassembly' }.

I've linked it before, but the Wasm ESM integration spec issue is here - WebAssembly/esm-integration#42.

@GeoffreyBooth that import('third-party', { assert: { type: 'javascript' } }) would stop the ability for a dependency to change implementation details between JS or Wasm seems overly restrictive to me, when there is zero practical benefit. @weswigham's "type": "any" would be preferable here. We shouldn't follow generalization for generalizations sake when it harms platform interop scenarios. For this reason I am likely to block that PR until these spec questions are clarified.

[bmeck]

Currently though, the behavior seems to be equivalent regardless of if it is the previous Symbol based approach for ambiguity or javascript, does this change any implementation requirements?

that import('third-party', { assert: { type: 'javascript' } }) would stop the ability for a dependency to change implementation details between JS or Wasm seems overly restrictive to me, when there is zero practical benefit.

I once again express extreme skepticism around being able to do this swap without a JS glue facade.

"type": "any" would be preferable here. We shouldn't follow generalization for generalizations sake when it harms platform interop scenarios. For this reason I am likely to block that PR until these spec questions are clarified.

As stated in other parts of this discussion ambiguity and collisions cause need to model various different kinds of resolution when multiple cache keys are involved.

If there is a desire to block nodejs/node#40785 for now, please do so explicitly.

[GeoffreyBooth]

As the PR is already just about ready to merge and everything within it is experimental and/or flagged, my preference would be to land it and modify as needed later when these spec questions resolve. It would be a waste of effort to let the PR get stale and need to essentially be redone months or years from now when the committees settle on answers to these questions; I’d rather just open new PRs when that time comes to modify Node per whatever the decisions end up being, and the flags/experimental status can remain while there are still open questions. WebAssembly/esm-integration#42 is already over a year old; it might be that landing this support in Node does more to push that discussion toward resolution than us remaining blocked by that lack of progress.

[GeoffreyBooth]

I just found tc39/proposal-import-attributes#116; @weswigham in the future please tag us or add a comment to this thread when you’re adding something elsewhere that’s relevant to this discussion.

[GeoffreyBooth]

I dug a bit deeper into the Chrome/Blink history around type: 'javascript'. I found this PR: Add tests for assert type: 'js' and type: 'javascript', which added tests that those types would throw rather than succeed.

I opened https://bugs.chromium.org/p/chromium/issues/detail?id=1270635 as a bug report. Hopefully their team will comment on it with their view as to whether erroring on type: 'javascript' is intentional or a bug.

[GeoffreyBooth]

I opened whatwg/html#7342 to ask the HTML spec team what their intentions were regarding type: 'javascript'.

[GeoffreyBooth]

With regard to the history online around whether type: 'webassembly' (or some other string) is likely to be required, there are various mentions on issue threads but the most relevant seem to be these:

WebAssembly/esm-integration#42

Only four comments, the last of which is this:

Do we envision the capabilities of Wasm and JS to diverge? If so, requiring an assertion might be prudent.

https://github.com/tc39/proposal-import-assertions#worker-instantiation and following sections

The import assertions proposal itself has repeated references to type: 'webassembly'. That’s where I got the idea from.

tc39/proposal-import-attributes#19

This thread seems inconclusive.

---

If there are other places online where the topic is discussed, especially if those threads seem more authoritative than any of these, please share them. I haven’t found anything else that more directly discusses the topic, or has more authority. These links are the basis for my conclusion that the most likely outcome of the spec discussion is that type: 'webassembly' will be required, and that 'webassembly' is the most likely string.

The decision hasn’t been made yet, obviously, so you all are free to comment on these threads and persuade those teams otherwise if you’d like; and I would happily update Node’s implementation to align. That’s why this is flagged.

[GeoffreyBooth]

@guybedford and I chatted about his concerns. We agreed on the following:

• Guy wants to see further feedback from stakeholders in the Wasm ESM integration proposal as to whether Wasm will want to use or require an explicit assertion. The goal is to receive guidance by December 1.

• Guy is okay with the type: 'javascript' parts landing. We both would prefer a more-generic name for this, such as type: 'executable', but that would require a change to the HTML spec and we’re not going to hold up this PR over that.

• If we know nothing more by December 1 then Guy would be okay for us to land the PR with Wasm being treated as the same as builtins/CommonJS/ESM, where it’s implicitly the default type. We would not land with type: 'webassembly' unless we hear back from other sources that the explicit non-JavaScript type is likely to be required. (And if it seems likely to be required but a different string, we’ll adjust to use a different string.) The goal is to land this PR in one form or another before it gets stale.

The consequence of treating Wasm the same as JavaScript is that we wouldn’t have the ability to treat Wasm differently than JavaScript in the future in terms of security privileges. Guy thinks this is okay, as the two platforms are inherently equal in their security model and we shouldn’t imply otherwise via type assertions. Treating Wasm imports as the default type also means that Wasm’s own internal imports of other Wasm files would not need assertion types.

The question is whether other interested parties like the HTML spec and browser folks would insist that Wasm get its own type, whether 'webassembly' or 'wasm' or something else, or if they’re okay with it sharing 'javascript'. If the Wasm folks want 'javascript'—or in other words, no assertion required, hence the desire to rename 'javascript'—and the web folks want some other string, then we have a conflict that those groups will need to resolve and the question would become what should Node ship in the meantime. The entire Wasm import implementation is flagged so it’s easy for us to change at any time, if what we choose now ends up being different than the final decision. I want Wasm to map to some type, though, both because every module format should have a valid type and because that makes the implementation simpler; so even if the broader consensus isn’t reached by December 1 I’d like for us to agree on some tentative type for Wasm that Node can use in the interim.

[bmeck]

I have no desire to conflate type:javascript and WASM. Having a different key or value seems fine but I'd rather just block trying to treat 'javascript' as covering both MIMEs. The whole discussion in TC39 and notes are about this being a MIME check and WASM is not going to be under the 'javascript' MIME group I would assume. If the goal is to allow WASM to be imported w/o an assertion changing the implicit MIME check should be possible without breaking since the assertion is never exposed to user code on the web. So even if the default is currently javascript it could become executable or even something like javascript||wasm w/e the value is.

[bmeck]

Implied above, is that for Node it would be a breaking change if loaders are non-experimental and exposing the value.

[GeoffreyBooth]

Fortunately for us, both Wasm imports and loaders are experimental and will remain so for quite a while yet. So I don’t think it makes much of a difference what type we pick for Wasm; it could be type: 'experimental-wasm-type-this-will-change-youve-been-warned' if we want. Or we could add code that throws when Wasm is imported with an assertion (of any type) though I’d rather not build that useless special code path just because we can’t reach consensus.

I think if we can’t get any indication from the HTML side what string they’d expect as Wasm’s type, like webassembly or wasm, then maybe we should default to javascript for now as the least bad placeholder option until HTML makes their intentions known. That would appear to users as no change at all, as their import statements would continue working without needing an assertion. And we would just keep Wasm imports flagged until the spec situation is known.

[bmeck]

I think that the symbol is a less bad option personally, but won't block as long as we note here that it is likely to change and give a warning when wasm is loaded with the assertion.

[GeoffreyBooth]

Update: the HTML spec team has confirmed that it wasn’t their intent to allow type: 'javascript' to be explicitly specified. The Chrome implementation is correct, and the wording of the spec is wrong. They opened whatwg/html#7350 to update the spec.

I’m going to update the PR accordingly. After those updates, only user-visible change from the PR will be that now we will print an experimental warning when import assertions are used, which only can be used for JSON. The import of JSON feature itself was experimental, so using its flag was already printing an experimental warning, so even this change is very minimal.