Apple Pencil "zDistance" and Continual Events

[PixelDough]

Currently, touching the screen with the pencil triggers an "InputEventScreenTouch" event, which contains a Vector2 position. If "Emulate Mouse From Touch" is enabled, it also gets the tilt and pressure values.

When reading Apple Pencil input in UIKit and SwiftUI (and probably Objective-C although I lack experience in it), there is a property called "zDistance" which is the normalized distance between the screen and the maximum pencil distance.

My proposal is that the zDistance value be added in to either InputEventScreenTouch or InputEventMouseMotion, and any zDistance less than 1.0 should emit input events. This way the Apple Pencil's location can be read when not touching the screen, in order to have objects in game react to being hovered, and optionally react more based on hover distance.

I believe some desktop pen tablets report the z position of the stylus, so I feel like including it in InputEventMouseMotion makes the most sense. And considering that InputEventMouseMotion is the only way to get stylus pressure and tilt from the Apple Pencil, it would make the most sense to include the zDistance property in it as well.

Another option would be to add an entire new input event type specifically for Apple Pencil, although I feel like it makes more sense to follow the standard that's already been established with InputEventMouseMotion.

It would be bad to change the position values of the input events to be Vector3's as that would break existing code. Adding it as an alternate property (call it "zDistance" or "height" or something to that effect) would enable developers to read the value if they so choose.

I think it's really important to have from a UX perspective. The pencil is used similarly to a mouse, and a mouse can "hover" over buttons. But the only way to hover over things currently with the Apple Pencil is to actually be pressing on them, which is a different action entirely to hovering. It also opens the door for some new types of gameplay that take advantage of the zDistance value, such as games that require you to always be within hover distance while avoiding walls, or using that distance value for some kind of pseudo-3d gameplay

[bruvzg]

We can add it, but not sure if it's supported on any other platforms, in most cases it is only possible to get "in range"/"in contact" binary states not a specific distance. We probably should also add "rotation" support as well.

[PixelDough]

Yeah, I think in most cases on other platforms it would just return 1 or 0 based on the distance (1 being no contact, 0 being contact).

I suppose this really is two different but related requests. When iPad is used as a display for a mac, the Apple Pencil is read as pen-tablet mouse input. But running a game directly on the iPad doesn't detect hovering as mouse motion. I think if you're going to add the functionality to detect the Apple Pencil hovering on iPad, you probably already have the zDistance value and it would be very nice to make that property accessible somewhere. But I think the more important of the two is definitely getting the hovered Apple Pencil sending InputEventMouseMotion events

[Calinou]

But running a game directly on the iPad doesn't detect hovering as mouse motion.

Check the values of Emulate Mouse from Touch and Emulate Touch from Mouse in the Project Settings and try toggling them.

[PixelDough]

I tried both of those options (and combining them) and it still didn't work