NodeMaterialObserver: Force refresh for video textures.

[Mugen87]

Related issue: #31330 (comment)

Description

Using video textures in WebGPU seems more complicated than in WebGL since it is not possible to share an external texture. We have to recreate the bind group and the external texture for each render item. For material properties, we can enforce this in NodeMaterialObserver.

However, if video textures are used as spot light maps, there is no obvious way to enforce a refresh. Hence, I have removed the video texture from the webgpu_lights_projector demo. I think we can only properly support them on material level. And even then they are a bit problematic for performance in WebGPU since we can't share bindings in the same way like with normal textures.

I suggest to state in the documentation that video textures are not supported in SpotLight.map.

[github-actions]

📦 Bundle size

Full ESM build, minified and gzipped.

	Before	After	Diff
WebGL	338.27 78.93	338.27 78.93	+0 B +0 B
WebGPU	558.52 154.58	558.88 154.69	+361 B +105 B
WebGPU Nodes	557.45 154.37	557.81 154.47	+361 B +107 B

🌳 Bundle size after tree-shaking

Minimal build including a renderer, camera, empty scene, and dependencies.

	Before	After	Diff
WebGL	469.61 113.66	469.61 113.66	+0 B +0 B
WebGPU	634.18 171.62	634.54 171.73	+361 B +113 B
WebGPU Nodes	589.31 160.95	589.67 161.11	+361 B +160 B

[sunag]

I'd like to analyze this in more detail. Currently we can take a uniform to an isolated group using uniform().setGroup() then we could share that group between materials using sharedUniformGroup(). But I didn't do the tests using VideoTexture.

[Mugen87]

I'm not sure this fixes the issue.

What irritates me is that the scope/lifetime of an external texture is so short. I've assumed calling importExternalTexture() and using it with a new bind group is valid until the current requestVideoFrameCallback() ends. I was surprised to see the WebGPU warnings during a single render call.

@greggman Can you enlighten us, please? 😇

In the fiddle, we use the same external texture and bind group for two plane meshes. The browser complains now with the warning: "External texture [ExternalTexture (unlabeled)] used in a submit is not active.".

https://jsfiddle.net/86t42dwk/

[greggman]

Without looking, importExternalTexture is valid until the task it's called in exits, regardless of which task. In other wrods, if you call it in requestAnimationFrame it's valid until your requestAnmimationFrame callback returns. If you call it in requestVideoFrameCallback it's valid until your requestVideoFrameCallback returns. If you call it in setTimeout it's valid until your setTimeout callback returns. If you call it inside mouseevent, it's valid until your mouseevent handler returns, etc...

[greggman]

And to be more clear, you must call importExternalTexture, put it in a bindGroup, use it in some pass, and call submit with the encoder that uses it, all in the same task.

[greggman]

Also, not that this explains anything but it seems every other frame the code calls importExternalTexture twice?

https://jsfiddle.net/1s97uLj2/

[Mugen87]

Okay, things are more clear now.

A new video frame is not available in every frame. When no new video frame is ready, the material observer does not trigger a refresh since it thinks the video texture has not changed. However, that will not trigger the required call for importExternalTexture() and creation of a fresh bind group. The PR is fixing exactly this issue.

There is one thing we can optimize: Use importExternalTexture() just once per frame for each video texture.

[Mugen87]

@sunag External textures are now cached per frame id. I've also made sure the video texture refresh is only forced for the WebGPU backend.

I'm curious to see if you can improve the approach somehow via sharedUniformGroup().

[greggman]

It sounds like you solved the issue?

One more caveat. While the external texture needs to be used in the same task, if you call importExternalTexture you might get back the same texture as before, in which case you don't need to create a new bind group.

In other words, in pseudo code

   let prevTexture;
   let bindGroup;

   rAF/rVFC/setTimeout/mouseEvent/etc...
     const texture = device.importExternalTexture(...);
     if (texture !== prevTexture) {
       prevTexture = texture;
       bindGroup = device.createBindGroup({
         entries: [
           ...
           { ..., resource: texture },
           ...
         ],
       ]);
     }

I don't feel like this really matters but just for completeness it seemed good not to leave it out.

[Mugen87]

It sounds like you solved the issue?

Yes, I think it's just a matter of how we organize our code now^^. The importExternalTexture() counter from the fiddle was very helpful, though 👍 .