About off-screen rendering

[torokati44]

Introduction

A while ago, I started experimenting with off-screen rendering in Ruffle.
This will be needed for some features, such as:

• the BitmapData.draw() method
• the cacheAsBitmap property
• applying the filters in the filters property
• rendering with any blend mode other than "normal"

As the first is the one requiring the least amount of extra supporting work, that's what I started with.

Internals

See the four latest commits in this branch for my changes so far: https://github.com/torokati44/ruffle/commits/bitmapdata-draw-noconflictrebase
There have been some refactors done to the wgpu backend on master since then, so I couldn't easily rebase them, but the idea is there.

The new [begin|end]_frame_offscreen methods would basically turn the renderer backends from a 2-state FSM (either there is a frame in progress, or not), into a 3-state FSM (no frame, onscreen frame, offscreen frame). This could perhaps be made explicit, and some asserts could be put in place, to ensure proper sequence of calls.
My initial conclusion was that there shouldn't necessarily be a need for both an onscreen and an offscreen frame to be in progress at the same time.

Backends

The implementation for the wgpu backend was not that difficult, thanks to the already existing similar functionality for the exporter package. I still had to fix some small bugs, and in the end struggled with asynchronity issues(?) when trying to properly resize the render target between "frames".

As I have no experience with either of the web backends, I left the new methods as unimplemented stubs in those for now, but I imagine creating a <canvas> that is not in the DOM (or maybe it's just hidden) could do the trick; or perhaps https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas would be needed.

At the moment the image data is always copied back to a CPU-side array. Mostly for simplicity reasons.
This will be necessary for some things though. BitmapData always stores its data in "CPU-side memory" as it is now.
Then, for example, until there are backend-native implementations of filters (maybe as shaders), applying filters will have to be done entirely by the CPU.
In the future, this could be optional, because for simple blend modes and the like, the actual data is not needed outside of the backend.

Results

The commits linked above have already, at least partially, fixed #3193 and #2296.
Also they make the character creator on https://www.dolldivine.com/star-wars-an-avatar-creator.php work better.
I also threw together a quick test that seemed to work just fine:

Questions

• Is this approach (3-state FSM) a good way to go about it?
• Would this work for the web backends as well?
• Is anyone able and willing to help me finalize this to the point that at least BitmapData.draw() works acceptably well? The main remaining issue is the proper resizing of the offscreen render target between frames.

[torokati44]

This also makes The Binding of Isaac look better, at least the parts covered by the offscreen buffer:

Before:

After:

[torokati44]

Also, at the moment, in BitmapData.daw(), the newly rendered content is blended on top of the existing contents in a simple (and most likely slow) per-pixel loop on the CPU side.
This could be avoided if the existing contents were blit[ted?] into the render target first.
Maybe even a new blit_image method could be added to the render backend trait, as this could potentially be much faster - not to mention more likely to be pixel-perfect - than rendering a textured quad.
This operation would also be useful in the future for rendering the proxy image (when caching as bitmap, or after applying filters) of display objects.