Precise input timing information

[cryslith]

What problem does this solve or what need does it fill?

The application I'm concerned with is rhythm games. Currently it is not possible to determine the time at which an input event (such as a keyboard press) occurred with precision better than 1 frame. However, rhythm games require input timing information which is more precise than this. For example, the ECFA Fantastic window is 11ms long, and the osu! 300 window can be as short as 13.33ms. On the other hand, a frame at 144fps is 6.94 ms long, while a frame at 60fps is 16.66 ms long. This means that the error incurred by measuring the time of an input precise to the next frame could be 50% or more of the actual timing window, making it impossible for players to reliably input in the window.

Besides rhythm games, precise input timing information is also applicable to other issues, such as #5984 and #6183.

What solution would you like?

Add Instant::now() to events processed in the winit event loop and the gilrs event loop. Additionally rewrite the gilrs integration so that gamepad events are polled continuously (like winit events) rather than once per frame so that these timestamps are accurate.

What alternative(s) have you considered?

Use timing information from the OS passed through by winit and gilrs, instead of measuring it ourselves using Instant::now() in the event loop. This has several issues:

• OS timing information may be unreliable. For instance, some OS timing information is precise only to the millsecond, which might be suitable for rhythm games but not for other uses of timestamps. They might also be nonmonotonic etc.
• winit doesn't currently expose timing information from the OS (Input event timestamps rust-windowing/winit#1194)
• gilrs timestamps are expressed as SystemTime, which could lead to issues. For instance these timestamps aren't monotonic and are affected by system time changes.

Additional context

[maniwani]

Just a couple notes because we've actually been building towards a solution to this for a while.

Add Instant::now() to events processed in the winit event loop and the gilrs event loop.

This is the approach we're going for, but (currently) any timestamps taken in the winit event loop are likely to be aliased to the Instant a frame ends because the app updates inside the event loop. The loop can't timestamp events while the app is updating. Instant::now() seen by the loop is likely much later than the true Instant an event arrived.

#9122 will remove this obstruction, but until then these timestamps won't be much better than time.last_update().

It'd be best if winit propagated OS timing since #9122 can't help wasm32, but someone has to investigate if the timestamps come from monotonic clocks (or can be configured to come from them) and if they have 1ms accuracy (not just resolution) or better. That does seem to be the case based on this comment and my own reading, but I'm not sure. (Getting 1ms accuracy from Windows is just a little complicated¹.)

Additionally rewrite the gilrs integration so that gamepad events are polled continuously (like winit events) rather than once per frame so that these timestamps are accurate.

gilrs internally spawns a thread to run its event loop (except on wasm32) and that event loop is collecting events constantly and forwarding them through an mpsc. It's just the app only drains the mpsc receiver once a frame. Draining the receiver once a frame is the correct thing to do though. There are special cases², but rhythm games typically aren't exceptions.

I think we should upstream changes to gilrs so that either its loop captures monotonic timestamps or the API exposes a way to construct the event loop thread manually.

IIUC gilrs events report SystemTime because its evdev dependency on Linux does. libevdev can set its clock source to CLOCK_MONOTONIC, so the crate probably did it because there's no Instant epoch or constructor.

Footnotes

1. The clock source of GetMessageTime is usually only accurate to ~16ms, but it can be changed to 1ms using timeBeginPeriod. But timeBeginPeriod has significant caveats on builds older than Windows 10, version 2004. ↩

2. XR apps want the renderer to read events that occurred in the current frame to do things like "late latching" and "reprojection". ↩

[cryslith]

@maniwani Thanks very much for the detailed note! I'd be happy to help out with this effort somehow - what would be useful for me to work on?

[maniwani]

@cryslith, I think we need help adding timestamp support to events in general, only the "accurate" part is blocked on rust-windowing/winit#1194 or #9122. For rhythm games, I assume there's probably more to it than just having timestamps? Do we also need to translate between "audio time" (e.g. DSP clock time, track time) and "app time" (time.elapsed())? Stuff like that.

For gilrs, I'd like to ask them to make either one of those changes I described (or both), but if we need something sooner, we could go with your plan and funnel their events through another thread that calls Instant::now(). It'd just be extra overhead.

[cheako]

1. This should ffect most every game. Should "coyote time" be tied to or a multiple of FPS/ticks? Lots of games would have some form of skill-based time limit and players shouldn't be floating on air for an indeterminate period while the GPU draws a new frame. When ppl talk about frame perfect tricks they are usually talking about NTSC/PAL and not trying to time things to the random noise of modern games.
   • Perhaps the play is to signal at a given specific time, the last chance to railyard hump the event b4 jumping is flagged as impossible. This notably doesn't help for the OP, but would be enough for many of the cases I can think of.
2. Timing is a std part of the Linux Input Event ABI, it would be a shame to learn it's a lie. "OS timing information may be unreliable." how much lifting is the "may be" doing here? Is there evidence of this, is it driver/device specific issues? Is it instead that "it's possible" or "could be?"