Video as a texture?

[bes]

Hello Bevy!

Does anyone know if I can load a video into bevy and get the playing video as texture(s)? I tried searching on a search engine as well as here but couldn't find anything.

Thanks.

[pkupper]

There is an open issue for video to texture streaming: #5221

Unfortunately there doesn't seem to be any plugin yet that supports this. You'd have to build something yourself using a video decoding crate such as gstreamer or ffmpeg-next.

[pkupper]

I just created the following demo application that uses ffmpeg-next to decode the video and displays it as an ImageBundle in Bevy:
https://gist.github.com/pkupper/108eb8a712f479ecfdb1eaf9b86cd128

Perhaps you can use this as a starting point. As-is it just displays one video frame for every Bevy frame and stops on the last frame.
If you want to have more control over playback that would require some additional work.

I might turn this into a plugin at some point.

[bes]

That's very cool!

If I read it correctly, it's copying the frame from CPU to GPU on every frame? I checked out #5221 and followed the linked issues, and it seems there doesn't exist any APIs for hardware decoding of video?

Do you know if there is an API in bevy to use an OpenGL texture created from another library like GStreamer? Or is that a hard problem to solve? I guess they have different OpenGL Contexts?

[pkupper]

I believe ffmpeg does use hardware acceleration by default, so it is possibly even going from GPU to CPU to GPU every frame which is definitely not ideal.

I don't know how hard it would be to go from raw OpenGL handles to wgpu types and back and if that would force you to use the OpenGL backend in wgpu. Possibly related wgpu issue: gfx-rs/wgpu#965

[bes]

AFAIK wgpu is supposed to be used by Firefox right? So the texture sharing would need to be solved, right? Because today you can efficiently use a video element as a texture source for WebGL.

At least one can hope :)