Should wasi:surface be split out into a separate proposal?

[lukewagner]

Having wasi:webgpu correspond directly to the existing WebGPU Web API makes a ton of sense. But the functionality in wasi:surface which includes the keyboard and mouse input is wayyy outside the scope of WebGPU. There are Web APIs for describing input sources, of course, and indeed wasi:surface/surface.key references https://w3c.github.io/uievents-code/#code-value-tables, but since input is defined by wholly separate APIs that are useful and usable independently of WebGPU, it seems like input functionality should be a separate proposal that is published, versioned and advanced separately from wasi:webgpu.

Incidentally, there have been a number of iterations on input on the Web Platform (indicating that it's a subtle and nuanced domain); see Pointer Events, Touch Events, Pointer Lock and the Keyboard API. Back when I worked on the Mozilla Games program, talking to gamedevs, some of these subtle API details mattered a lot, so I think it'd be a good idea to involve an expert in this area to help us scope out a proper "wasi:input" package and the set of Web APIs we should mirror into it. Or maybe there are other popular input APIs we should mirror that make more sense outside a browser? Inventing our own thing from scratch seems risky unless we have some significant input API expertise of our own. In any case, there will be some non-trivial design work, so it definitely seems appropriate to decouple from wasi:webgpu.

[MendyBerger]

Hi @lukewagner, thanks for filing this issue!

Agreed that input is a nuanced and detailed domain, and its precisely for this reason that we decided to model it similar to the web, but we tried to leave out anything that's a bit more niche and not needed by most apps. That is, we included pointer-events as it abstracts over mouse, touch, pen, as well as keyboard. But left everything else out, as pointer and keyboard should cover the vast majority of apps.

That's not to say the the other items aren't important, they absolutely are. But building a full windowing system is an incredibly complex task, that requires a deep understanding of every operating system out there, as well as how they each relate to security/sandboxing/finger-printing. So we opted for a basic surface, modeled over something that already has multiple implementations on multiple operating systems (the web), but we were very careful to make sure there's room for a newer, more complete, full windowing proposal.
If I had to guess, I'd say that there's literally years of work to designing such a system from scratch.

We'd love to see people with domain expertise getting more involved! Do you have anyone in mind?

Re breaking out the proposal: I don't have a strong opinion on that, either way works for me. We did have a couple of conversations about this with people in the community before we moved to phase 2. Ultimately, we went with having separate packages in one proposal, but I'm open to changing that. Just don't wanna be changing proposal structure every few months, so maybe let's first make sure that there's strong consensus on the matter.

Thoughts?

[pchickey]

Right now I think we are best served by collecting stakeholders and experts to collaborate on the various packages here all as part of one proposal, until we see that its no longer serving the process.

To me there are two reasons to break out a package from this proposal into its own proposal:

• it has a different enough set of stakeholders, experts, and users that they want to have separate meetings, champions, bug tracker etc.
• it is proceeding through the stages at a different rate than the rest of the proposal

If someone can make a compelling argument that one or both of those have created a problem for wasi-gfx or wasi-surface, we can split packages off this proposal. I don't want to do so, however, on the presumption that one or both of those may become a problem in the future.

[lukewagner]

Scanning wasi:surface, I didn't realize it was based on Pointer Events (viz. because there was no mention of Pointer Events and "surface" is not a concept in the Pointer Events spec), and so I assumed it was an ad hoc interface, which would require a lot more evaluation than an interface that is 1:1 with a standard Web API. If wasi:surface is indeed based on Pointer Events, then it would be good to have the correspondence laid out a bit more clearly in names and doc-comments, as we've done with wasi:http and (I assume) wasi:webgpu. E.g., wasi:surface doesn't seem like the right name.

More generally, we didn't add a key-value-store or config-store to wasi:http (even though many proxies want such functionality and proxy-wasm indeed bundled such functionality); instead we have separate wasi:keyvalue and wasi:config packages/proposals that describe these modular sets of functionality so they can be used modularly. Putting a subset of input events into a monolithic graphics proposal seems similarly conflating two distinct domains which are naturally going to evolve separately over time. Similarly, I don't think we should vote and publish them as one monolithic unit; we've been doing that on a WIT package level which then maps naturally to package versioning.

Thus far we've had one WIT package per GH repo, which suggests separate repos. I suppose that's not a hard rule, but I wonder if it messes with any of our automated processes for versioning/publication?

[lukewagner]

Also related to input:

While I've heard a number of obvious use cases for running wasi:webgpu outside the browser in a headless mode where there is no user input, I'd be interested to see the list of use cases which want wasi:webgpu and user input outside the browser, since it seems like there is a spectrum of possible use cases which suggest different interface designs. In particular, is the component trying to act like a whole GUI "app" or is it more like the logic associated with a single "widget" (or "control", in OLE parlance) or something else? If a single component instance can be associated with multiple surfaces (which the current WIT seems to suggest, unless I'm reading it wrong), then it's sortof neither, and so I'd be curious what the intended mental model is and what's the relationship between the component and the containing GUI non-browser host.

[MendyBerger]

@lukewagner yeah, the wit isn't documented well, and I didn't do a great job explaining it above. My apologies for that.
In fact, stabilizing and documenting the wit is likely going to be one of my main areas of focus in the next few weeks.

Just to try and clear up what wasi:surface tries to be:
In the early days of this proposal, we briefly thought of defining a full windowing interface, someone even put up a PR for that, but we quickly realized that designing such an interface will require an amount of expertise and bandwidth that our small group simply didn't possess. However, we still believed that we can build something useful for now, and make sure we keep the door open for broader proposals later.
So here's what we're trying to do now: we're trying to define the smallest useful surface/canvas/window/whatever-you-call-it that is still useful. We considered having just a literal surface that gets no user input events, but that didn't seem very useful. So we decided to steal pointer and keyboard events from the web. We didn't copy them as closely as we copied webgpu, but the semantics are - or should be - mostly the same.
With a surface, keyboard events, and pointer events, we believe that we cover most general apps. And by taking the semantics from the web, we avoid having to do our own bikeshedding.
So in short: wasi:surface is a surface api, with the addition of pointer and keyboard events.

As to what kind of apps we're trying to cover: we wanna cover both standalone apps, as well as widgets in existing apps. For now, we believe wasi:surface can cover both. We're actually hoping to start doing some experiments testing surface as a plugin/widget soon.

Might also be worth mentioning, that if a host app wants to provide wasi:webgpu to it's plugins, but wasi:surface is too basic for their usecase, they could easily provide a custom windowing system, and have that plug in with wasi:webgpu.

Does that clear it up at all?

[pchickey]

More generally, we didn't add a key-value-store or config-store to wasi:http (even though many proxies want such functionality and proxy-wasm indeed bundled such functionality); instead we have separate wasi:keyvalue and wasi:config packages/proposals that describe these modular sets of functionality so they can be used modularly.

Those packages are not yet ready to standardize, but wasi-http was. Packages that mature at different times should be in different proposals. Packages are the unit of software engineering modularity, proposals are a unit of standards process modularity. There is no reason to equate the two.

Putting a subset of input events into a monolithic graphics proposal seems similarly conflating two distinct domains which are naturally going to evolve separately over time.

These domains are closely related and the champion of this proposal is working enthusiastically on both. How different domains are is a totally subjective argument.

However, I strongly object to making changes to a structure that is working today on the supposition that something will happen to to separate them in the future. They are fine now, so let them be now. When it gets to be a problem, we can address it then.

Similarly, I don't think we should vote and publish them as one monolithic unit; we've been doing that on a WIT package level which then maps naturally to package versioning.

We de-facto voted for and published a large set of distinct proposals as one monolithic unit as we established the 0.2 release. There is no reason we can't vote for multiple packages as one proposal. The only reason to split them off is if they need to progress at different rates.

Thus far we've had one WIT package per GH repo, which suggests separate repos. I suppose that's not a hard rule, but I wonder if it messes with any of our automated processes for versioning/publication?

Splitting efforts among many GH repos has many adverse effects as well that needs lots of tedious tooling work to resolve. I haven't seen any evidence we can't make multiple packages in a repo and proposal work on technical grounds - any issues we have can and will be fixed, just like we worked around all the headaches of having the initial 0.2 set split among many repos.

[lukewagner]

@MendyBerger Thanks for all the background! Those goals and challenges make sense. Agreeing with and synthesizing these points:

In the early days of this proposal, we briefly thought of defining a full windowing interface, someone even put up a PR for that, but we quickly realized that designing such an interface will require an amount of expertise and bandwidth that our small group simply didn't possess.
[...]
So here's what we're trying to do now: we're trying to define the smallest useful surface/canvas/window/whatever-you-call-it that is still useful.
[...]
As to what kind of apps we're trying to cover: we wanna cover both standalone apps, as well as widgets in existing apps. For now, we believe wasi:surface can cover both. We're actually hoping to start doing some experiments testing surface as a plugin/widget soon.

I think the minimal thing we can get away with in the short term is removing wasi:surface from this WASI proposal and decoupling its advancement from that of wasi:webgpu so that wasi:webgpu can advance quickly. Usefulness in the short term is then achieved by expecting wasi:webgpu to be used by non-WASI interfaces that take responsibility for connecting wasi:webgpu to surfaces/widgets/canvases/whatever. This is business-as-usual for, e.g., OpenGL, which has the same diversity-of-embeddings situation that we have.

In particular:

• Headless applications would be able to simply include wasi:webgpu along with wasi:cli, wasi:http or wasi:messaging, and be off and running.
• UI applications that want to run portably both inside and outside the browser could start by using webidl:-namespaced WIT interfaces generated from relevant Web APIs (using your excellent work on all that) and then emulating the browser's behavior in outside-the-browser hosts. Because the mapping is direct, there is less to document and teach, and as much or little of Web API-defined functionality can be pulled in, since we're not trying to standardize it (yet, see below).
• To your final point, outside-the-browser native UI frameworks (e.g., Qt, Reactive Native, WPF, GTK+, ...) will naturally have different opinions on what the ideal shape of an embedded wasi:webgpu-using component is w.r.t how components are created and connected to widgets, and these can each be developed as-needed with WIT interfaces in their own namespace to explore this design.

Over time, I think a WASI proposal focused on input (orthogonal to WebGPU vs. 2D vs. ...) that supports embedding in the browser, fresh whole-app wasm runtimes and existing widget frameworks would be valuable and is probably inevitable but, as you said above, doing it well is quite a significant undertaking and will require more time and stakeholders than we currently have.

This approach takes advantage of a bunch of the hard work we've done to decouple WASI from the underlying WIT and bindings-generation tooling so that non-WASI interfaces don't have to start from scratch and thus embeddings using approaches 2 or 3 above should actually have an easier time in the short term getting something working.

[MendyBerger]

I think the minimal thing we can get away with in the short term is removing wasi:surface from this WASI proposal and decoupling its advancement from that of wasi:webgpu so that wasi:webgpu can advance quickly.

So far wasi:surface has not been holding wasi:webgpu back at all. Should that change in the future, I'd be a lot more open to decoupling them.

I strongly believe that wasi:webgpu would lose a great deal of value if it couldn't be used for platform agnostic UI.
I also believe that we can get wasi:surface to a place where it covers a wide variety of UI use cases, and that we can do it in a reasonable amount of time, without it holding back wasi:webgpu.

---

outside-the-browser native UI frameworks (e.g., Qt, Reactive Native, WPF, GTK+, ...) will naturally have different opinions on what the ideal shape of an embedded wasi:webgpu-using component is w.r.t how components are created and connected to widgets, and these can each be developed as-needed with WIT interfaces in their own namespace to explore this design.

I think that's true even if we let wasi:surface progress.

[lukewagner]

So far wasi:surface has not been holding wasi:webgpu back at all.

My point with the above is that, while I think it's a good idea to advance wasi:webgpu in the short term to Phase 2, my impression is that wasi:surface needs more design exploration with more embeddings to validate before advancing to Phase 2 because UI is a domain that has doesn't have a clear "just convert this standard to WIT and you're good" story that we can leverage like WebGPU. Particularly when multiple surfaces are involved, there are I believe a number of subtle host/guest-interaction design questions. One example is lifetime and avoiding cyclic leaks, which in the UI domain, seems to be the problem you have by default if you don't proactively avoid it; most popular UI frameworks simply rely on cross-guest/host GC.

If wasi:webgpu is Phase 2 but wasi:surface isn't, there are a number of options for input-requiring runtimes:

• Keep wasi:surface at some pre-phase-2 -draft version while continuing to experiment, prototype and collect more outside-the-browser embedding experience.
• Use non-WASI WIT interfaces derived from Web IDL and implementing those in the outside-the-browser runtime and version based on the WIT-generator tool (since the Web IDL shouldn't change).
• Define a custom-fit WIT interface in your own namespace and release/version however you like.

This is similar to what other runtimes have done in the past for cases where the WASI standard wasn't ready to advance due to hard design questions (e.g., wasi-sql). I do want to say though that I do very much appreciate your work driving this all forward, so I don't mean this as a criticism of the work that has been done; it's just that, imo, UI is a complex domain requiring more overall design consideration in a WASI context than WebGPU, at least to meet the Phase 2 bar. But that is ofc just my opinion, so I'm happy to discuss in the broader WASI SG.

[seanisom]

Hi! Lots of thoughts. I'll try to keep this focused to the most relevant points:

My point with the above is that, while I think it's a good idea to advance wasi:webgpu in the short term to Phase 2, my impression is that wasi:surface needs more design exploration with more embeddings to validate before advancing to Phase 2 because UI is a domain that has doesn't have a clear "just convert this standard to WIT and you're good" story that we can leverage like WebGPU.

Not to be pedantic, but wasi-surface, wasi-webgpu, and the entirety of the wasi-gfx proposal is already Phase 2. We had discussed with many members of the BA and WASI subgroup across many meetings gaining broad consensus both for the rename to wasi-gfx encompassing all packages under a single proposal, as well as meeting the requirements of Phase 2. wasi-surface has a reference implementation that is already useful (below), but even if that did not exist, I am not convinced that anything has changed since the Phase 2 vote regarding readiness as described, and this is the first dissent from that consensus that we have noted (so, let's absolutely discuss!). To @pchickey 's point, we brought up in the discussion for the vote that it is likely we could split off proposal in the future if they are blocking each other, but we are not close to that being the case yet, unless we wanted to quickly advance wasi-webgpu to Phase 3 independently.

While I've heard a number of obvious use cases for running wasi:webgpu outside the browser in a headless mode where there is no user input, I'd be interested to see the list of use cases which want wasi:webgpu and user input outside the browser, since it seems like there is a spectrum of possible use cases which suggest different interface designs.

We have seen the opposite - we have only seen one potential use case for a headless GPU renderer that didn't shake out due to myriad complexity of Nvidia drivers with windows servers across cloud VMs. However, we have seen significant interest in running fully-GPU rendered GUI apps on desktop. This primarily stems from WGPU and Bevy, which both provide a Wasm browser runtime and a native desktop app target with a simple Window controlled by winit.

The goal is to provide a largely compatible runtime environment so these existing apps can use a WASI target to run in Wasm on Desktop. There are use cases for plugins / widgets / SubRects etc, but they are far less well-defined. As of now those are things that largely would need additional host<>guest interaction than wasi-gfx could provide out of the box, so they are not (at least yet) explicitly in-scope.

Usefulness in the short term is then achieved by expecting wasi:webgpu to be used by non-WASI interfaces that take responsibility for connecting wasi:webgpu to surfaces/widgets/canvases/whatever. This is business-as-usual for, e.g., OpenGL, which has the same diversity-of-embeddings situation that we have.

I strongly disagree these concerns should be non-WASI. This would break all of our existing examples (see the talk from WasmCon), the Bevy integration, the WGPU integration, and our graphtime host, which we have discussed merging into Wasmtime with their maintainers as things stabilize.

It's true that legacy OpenGL through adjuncts like GLFW and even modern DirectX through DXGI separate these concerns. However, that doesn't really map to the Web and WebGPU, which requires a GPUCanvasContext to draw anything, even offscreen, and does not give the user control over swapchains or windows. Our goal is to keep the API like WebGPU, not like native graphics APIs from a user's perspective, because there is no "host" code other than the browser implementation itself.

UI applications that want to run portably both inside and outside the browser could start by using webidl:-namespaced WIT interfaces generated from relevant Web APIs (using your excellent work on all that) and then emulating the browser's behavior in outside-the-browser hosts.

I am not understanding this, perhaps you could clarify. An explicit goal of the wasi-gfx project is to allow code written against Bevy / WGPU / Dawn / Browser WebGPU JS run with no or very minimal (so could be macro'd) guest modifications on a WASI host. If you're just talking about shimming/adapting host-side code from WebIDL to native versions that is introducing an additional abstraction layer that would slow down adoption, but be workable. If you're saying guest code should change to explicitly call prefixed APIs, that's a non-starter.

... E.g., wasi:surface doesn't seem like the right name.

I kind of agree with this. Surface itself is actually a good name (it matches Khronos/Vulkan terminology, and we explicitly don't want to call it a Canvas given the massive API footprint of the Web's Canvas that we are not trying to recreate.

However, it ultimately does not make sense for a Surface to be responsible for input - that's combining the spirit of a Back Buffer with a surrounding windowing system, whereas the Canvas/Surface are solely supposed to be an opaque handle to an object containing said Back Buffer.

I'm open to solutions - these could all be split off into separate objects, and perhaps at some point even proposals, at the expense of additional layers of abstraction.

... UI is a complex domain requiring more overall design consideration in a WASI context than WebGPU

I agree, and explicitly don't want to be the person implementing an entire Windowing or UI system within WASI! But that's not the goal of wasi-surface at all - it is a simple interface for compatibility that works for likely 95%+ of GPU-drawn Desktop apps that need basic mouse and keyboard input (with accessibility coming) and back buffer support through a Window.

so it definitely seems appropriate to decouple from wasi:webgpu

We've spent 6 months discussing this and having the community align around the approach we ended up taking, so I'm trying to be open to and not defensive about changing this now :) If this makes the most sense, I'm happy to change it although I am concerned about additional proposal overhead from previous conversations with Pat/Dan/Bailey. I assumed we would someday reach this point, especially once there is a desire to actually build a windowing API. I'm also starting to think it makes sense to decouple surface from input/other window concerns. My strong resistance is to getting rid of wasi-surface or shifting guest-specific implementations onto hosts, or even more strongly requiring modifications to guest code.

[lukewagner]

Hi @seanisom, thanks a lot for the thoughtful comments and openness to discussion. I'm sorry for confusing "Phase 2" with "Phase 3"; I had meant to say "Phase 3" and have no argument against the current "Phase 2".

we are not close to that being the case yet, unless we wanted to quickly advance wasi-webgpu to Phase 3 independently.

I had been under the impression that folks did want to advance wasi:webgpu to Phase 3 so that it was included in 0.2 or 0.3 quickly. From what I've seen, inclusion in a Preview release is a point at which design decisions start to get "baked in" and harder to change later. If that's not the case and the plan is to keep iterating in the current draft state for a while longer, then you're right, there's no pressing issue.

There are use cases for plugins / widgets / SubRects etc, but they are far less well-defined. As of now those are things that largely would need additional host<>guest interaction than wasi-gfx could provide out of the box, so they are not (at least yet) explicitly in-scope.

That's useful to hear! Based on above comments and the current WIT allowing what seems like a dynamic multiplicity of external surfaces, it looked like plugins/widgets/SubRects were in-scope, and this is where my technical concerns are because this is where I think (and it sounds like you agree) there is a lot of diversity of frameworks and opinions on what a "widget" is as well as the permeating risk of cycles. I do actually think that in the long-term there is huge potential value for WASI to work on this much harder problem, but it'll just take a lot of work and require a lot more participating UI framework people than we currently have; this is what I strongly felt we needed to decouple.

So, combining the above with your other comment...

... E.g., wasi:surface doesn't seem like the right name.

I kind of agree with this.
[...]
I'm open to solutions - these could all be split off into separate objects, and perhaps at some point even proposals

... if widgets/subrects are out of scope and we're mostly just talking about a component that owns the rendering of a single app's whole top-level window, then it seems like we could perhaps significantly simplify and reduce the current set of interfaces, avoiding the leak/lifetime issues in the process. Just as a sketch (with suggestive placeholder names that we could bikeshed later):

package wasi:webgpu {
  interface webgpu {
    resource surface { // now in webgpu
      // no constructor; it must have been created by "someone else" (e.g., see below)
      height: func() -> u32;
      width: func() -> u32;
    }
    ... the rest of webgpu as is, but using 'surface', replacing 'wasi:graphics-context'
  }
}
package wasi:ui {
  interface toplevel-window {
    use wasi:webgpu.{surface};
    get-webgpu-surface: func() -> surface;

    variant event {
      mouse-up(...),
      mouse-down(...),
      ...
    };
    events: func() -> stream<event>;
  }
  world app {
    import toplevel-window;
    ...
  }
}

(For clarity, I'm using a single P3 stream of events, but the sketch could just as well be applied to P2.) But could something roughly in this shape work for your intended use cases, or is it too simplistic?

I strongly disagree these concerns should be non-WASI.

To be clear, I wasn't suggesting "non-WASI" ever; I was just worried about attempting to do something widget-y in the short term. But based on the above, it sounds like there may be no problem; perhaps double-so!

[programmerjake]

I think that surfaces shouldn't require webgpu, since some embedded systems have no gpu and yet still have a UI.

[lukewagner]

@programmerjake I agree. A fuller sketch I think would include multiple optional imports of different kinds of surfaces or perhaps define different worlds which non-optionally imported just one kind of surface.

[seanisom]

Thanks for the comments @lukewagner @programmerjake!

Bigger reply to come when I'm back on WiFi, but yes, wasi-surface or surfaces in general should not be required. We didn't make that explicitly clear above. I think this already works from our ONNX example, but I'm not sure it is spelled out anywhere in the design/docs. @MendyBerger can you confirm?

[programmerjake]

but yes, wasi-surface or surfaces in general should not be required.

well, a surface seems like the obvious thing to use for a UI since it combines both video out and input events. So, I think surfaces should work with the framebuffer interface too, not just webgpu. saying you don't need surfaces for video out because you can use frame buffers instead thereby allowing surfaces to require webgpu is imo a mistake since the input is also linked to the surface, preventing getting input when using a framebuffer.