Frame Invalidation is stacking

[unphased]

I found some interesting behavior.

When I have the devtools open, frame invalidation based on mouse movements (with orbitcontrols) seems to stack. For example, I have a mouse that toggles profiles, and one is set to 125Hz polling rate, and one at 1000Hz polling rate. This means that this will be the frequency with which the OS will feed input events to an application.

I noticed that something was weird because when my application changes state that affects the 3d canvas, it will render frames for a few seconds even when frame invalidation is turned on. I thought this was odd so I was tinkering a bit. I found that when interacting with the mouse to move the camera, after releasing the mouse, for some time it will continue to render out frames.

I realized that when I switch to an input device that has a lower 125Hz polling rate, since my monitor's refresh rate is close (120Hz), it does not exhibit this behavior! And if I change my monitor to 60Hz, it will do it again, but reduced compared to before. Starting to make sense... So anyway, after extensive testing, I did determine that indeed invalidate runs at mouse polling frequency, only when the devtools are open or when recording with the performance tab in chrome, it does not happen when the devtools are closed, or under firefox.

I have seen https://spectrum.chat/react-three-fiber/general/clarification-about-invalidateframeloop-invalidate~1c0c4a70-4df1-4eaf-b5db-4323e7b1b2cd in which it's made clear that the intent behind invalidate is such that it's marking it dirty so rendering the next frame is performed.

For whatever reason, it seems that this is

• not happening properly with OrbitControls when in chrome and devtools are open (???! seems like somehow the mouse rate is being throttled by chrome itself only when devtools isn't open)
• not happening with react state updates!

This is a screenshot from my app where I have added some performance.mark instrumentation in node_modules/react-three-fiber/web.js around the invalidate call, actually, although that isnt directly visible in the screenshot there, you can see where my mouse hovers (sorry no mouse cursor is shown but the popup is shown), this entire stack is react updating and it seems to be doing a whole lot of invalidate calls in there, and these actually do stack up, resulting in several seconds worth (hundreds and hundreds) of frames being rendered after the react operation is done. I'd like to do something about this since those invalidates are being called all while rendering is blocked so it really only needs to render either zero times or once after doing all the work, not hundreds of times.

I also have invalidate (in react three fiber) instrumented with this log:

console.debug("invalidate called", frames, state.current && state.current.frames);

which prints the amount of stacked frames!

Here is a video showing the logs with different rates of input:

2021-02-02.01-57-29-3.mp4

you can sorta see when it stops rendering because i did crank up the noise postprocess filter. Though, the compression I had to do to get it under 10MB sort of destroys that. Anyway you can see whats going on with the log output on the console.

The point is not that it's a bug that gaming low latency mice cause extra stacked renders when the events don't get coalesced, although it's not clear yet why coalescing fails to occur when the devtools are open in Chrome, it's normal that it happens. Here's some info on what the coalescing is about https://developers.google.com/web/updates/2017/06/aligning-input-events. The problem is with the queued number of frames continuing to render continuing to increment, by default, when the behavior should really just be marking it as dirty. I wish I was more familiar with what reasons exist for actually counting the number of frames to stack, then I could try to improve the code.

[unphased]

Looking at the code some more:

  if (state === true) {
    roots.forEach((root) => {
      const state = root.containerInfo.__state
      state.current.frames = state.current.ready ? state.current.frames + frames : frames
    })
  } else if (state && state.current) {
    if (state.current.vr) return
    state.current.frames = state.current.ready ? state.current.frames + frames : frames
  }

I would like to have a better understanding of the function of the ready state here. Since at least from my vantage point, the invalidate is being called a lot in the course of react rendering where, empirically, ready is true at least most of the time, and this causes the frames count to increase to a level that in my app (which is currently poorly architected and causes renders to happen every few seconds even while idle) really limits the benefit of frame invalidation (i.e. reducing heat and conserving battery, in the case when the app doesn't need to animate anything).

I've patched my code by changing this code to: state.current.frames = frames, and now I can develop my app on my intel macbook again without huge battery drain ;)

[drcmda]

invalidation doesnt render, it is a simple count, and if it's > 0 it will render once per frame, never more than once. it can double render (two frames, one after the other) to avoid some browser nonsense like minimize/maximize being a event late - so invalidate is not deterministic, but this is a dead-end, that's not the problem you're facing.

in your screenshot there is no indication of three rendering more than once, this is all react doing some intense churn and it's most likely caused by a bug in your app, not r3f. it looks like setstate in a loop to me or inside a event, which would not be good.

[unphased]

Thanks for looking at this. I do realize that my explanation is lacking and I did a poor job of emphasizing the core issue. I'll try again when I have more better information to share on this. But when I tested it, it's definitely the case that the counter just keeps incrementing whenever you receive input events at a frequency higher than monitor refresh (a very likely scenario on most machines, since refresh will occur at 60Hz and mouse updates come at 125Hz or faster) and on react renders (which may or may not be application/implementation bugs, fair enough).

I totally get that sometimes it's necessary to queue up more than one additional frame to render for various reasons. I've been doing some GPU picking related stuff and often we need to render twice (once just to get data out of the GPU and one more time to get that data rendered so it becomes visible). This does not change the fact that most scenarios (such as react rendering) we just want to assert the notion of invalidating the render state, rather than to add one to the counter of how many additional frames we need to render next. If you actually log that counter, you'll see in lots of typical scenarios that counter will rise very high when it totally doesn't need to. For example if we add a thousand objects in our scene with R3F it'll cause this counter to go to over a thousand, so now we'll render over a thousand frames that didn't require it.

But anyway, yeah i'm this territory that isn't a very popular one to play around in, being super miserly about rendering out frames. Things do become way simpler to just do the simple thing and render fresh frames indiscriminately. I still would like to express my deep gratitude that a tricky feature like this (frame invalidation) even is a part of the featureset of the library in the first place. It really is a mark of going above and beyond.