Call for possible collaboration

[kermitfrog]

Hi,

I'm the developer of a tool called inputMangler, which transforms input events on linux.
After a few years of other priorities I want to continue development (well.. rewrite it from scratch actually..).
As I like to avoid duplicate work, I had a look around the net to see if someone else started another project like mine. I found a few which at least do something similiar and, if you're mentioned at the end of this post, one of them is yours.

While all those projects seem to have more or less different goals and approaches, there still might be enough common ground for collaboration.
So this thread is about exploring possibilities to work together.

In the next post, I will write an overview of my goals. I invite everyone interested to do the same.

Afterwards we can compare those and discuss if it would make sense to
• put some base code in a common library
• merge projects (may be unlikely, but .. maybe)
• just share experience on strange input-related problems ;D

Links to the projects:
https://github.com/kermitfrog/inputmangler
https://github.com/sezanzeb/input-remapper
https://github.com/samvel1024/kbct
https://github.com/shiro/map2
https://github.com/rvaiya/keyd
https://github.com/snyball/Hawck
https://github.com/KarsMulder/evsieve

And the people that I hope will have a look at this after receiving a notification for being mentioned:
@sezanzeb, @samvel1024, @shiro, @rvaiya, @snyball, @KarsMulder

[kermitfrog]

Goal overview of inputmangler

Working in current version

• direct remapping of linux input events (keys, mouse wheel, etc.) for multiple devices
• hierarchy of presets, which activate depending on window class and window title (no need for manual change of presets)
• support for triggering combos (e.g. shift-a), macros, mouse wheel acceleration
• dbus interface (change preset, update config, create event, print current window information)
• localized key names (kind of - they're actually custom configured)

Planned

• Easy to use UI
• system service which uses something like policykit to detemine current user
• plugins

Nice to have if feasable

• handle more complex input like key combinations
• possibility to run external commands or sending dbus events on certain conditions
• config format that can easily be edited manually. This was a priority before and is working in the current version, but I would change it in favor of a cleaner format.

Frame

• linux first
• focus on reliability, performance and security
  • rewrite in Rust
  • experimental features that are in conflict with above requirements:
    • have to be explicitly allowed in service config
    • have to be explicitly activated by the user
    • trigger a warning in UI if activated

[sezanzeb]

Have you considered contributing to the existing ones?

I'll probably invest my time into maintaining the 2.0 (beta right now) release coming in February

[kermitfrog]

Have you considered contributing to the existing ones?

I am still considering that and its part of what this thread is about.
We all have certain ideas about what the program should do and how best to achieve that. So the question is: is there a project where I can realize my goals by contributing (this would clearly be the preferred way) or are the projects goals incompatible with mine?

Project descriptions tell something about what the current state is, but little about what is planned.

Having everyone summarize their goals and priorities would help a lot to clear this up.

[sezanzeb]

Ok

For input-remapper the current goal is to finish 2.0 up. The current work on that is happening on the beta branch: https://github.com/sezanzeb/input-remapper/tree/beta. After that pretty much any input can be mapped to anything. For example mouse movements to joysticks. It will feature an overhaul of the GUI to support all that without editing configs. After the release, people might discover bugs, since a lot of new stuff will be released.

See sezanzeb/input-remapper#177 for information about contributing, and https://github.com/sezanzeb/input-remapper/blob/beta/readme/development.md for some technical details.

Works:

• direct remapping of linux input events (keys, mouse wheel, etc.) for multiple devices
• support for triggering combos (e.g. shift-a), macros, mouse wheel acceleration
• dbus interface (change preset
• system service which uses something like policykit to detemine current user
• handle more complex input like key combinations
• config format that can easily be edited manually. This was a priority before and is working in the current version, but I would change it in favor of a cleaner format.

reliability

If input-remapper doesn't work, then it is usually because something is fundamentally broken or impossible as of now. But it seems to be quite stable during operation. There are tons of automated tests.

• Easy to use UI

I like to think it is

• linux first

input-remapper will never support anything other than linux probably

Somewhat works:

• hierarchy of presets, which activate depending on window class and window title (no need for manual change of

Via third party software: https://github.com/DreadPirateLynx/input-remapper-xautopresets. This needs to be individual for each Desktop Environment. There is no solution that works for all Wayland DEs. It's easy to do in X11 apparently.

• localized key names (kind of - they're actually custom configured)

This works for X11, gnome on wayland, plasma on wayland, but other DEs that run on wayland may not support it properly. Input-remapper has to rely on using xmodmap -pke to get that information, which is part of xorg. Right now this is not causing trouble.

performance

Not causing any issues, but CPU usage can go up to 5% on my computer during usage (on a single core). input-remapper-service has never been profiled properly, there might be potential for optimization.

security

Key logging is possible or a few minutes while the GUI is open. There is no way around that. Because information has to go from a privileged service to the unprivileged GUI via a pipe to record input. Other than that, I don't think input-remapper is leaking input anywhere during normal operation.

Doesn't work:

• plugins
• possibility to run external commands or sending dbus events on certain conditions

Because the daemon runs as root, which is a security problem if mappings trigger commands and challenging to sandbox properly to not cause problems. I'd like to avoid those things. Running external commands is often possible via the DEs settings, and probably sufficient for most users.

• dbus interface update config, create event, print current window information)

updating config is done via the GUI, which just writes to a json file

[kermitfrog]

Thanks for the info :)

Easy to use UI

I like to think it is

I tried your beta and think the UI has a solid concept and is well done (although some polish is needed - which is to be expected in beta).
However, it follows a different logic than I would like (Device > Preset > Mapping vs. Preset > Device > Mapping).

I find defining output combos difficult though. Autocompletion is a great idea, but recording the output sequence should be better in most cases.

Not causing any issues, but CPU usage can go up to 5% on my computer during usage (on a single core).

For comparison, I tried it on my computer and it goes up to 9 % for mouse movement and 12 % with a SpaceMouse compared to 1.3 % / 2 % with inputmangler, so there is clearly room for imrovement.
Are you using Cython?

[sezanzeb]

Are you using Cython?

Yes, not much difference with pypy once the jit compilation has started optimize it

I find defining output combos difficult though. Autocompletion is a great idea, but recording the output sequence should be better in most cases.

Have you seen the information on the botton of the output editor?

It was added for the purpose. If there is no device available to record the output the user wants it might get difficult to set certain mappings.

[sezanzeb]

although some polish is needed

You are very welcome to tweak it in Glade and to make a PR

Device > Preset > Mapping

and also to create a new issue to discuss this. Showing how the GUI would have to change, and explaining how the workflow of recording input would change would be helpful there :)

[KarsMulder]

My two cents: I think that the hardest part of a keymapper project is actually not the implementation, but the design.

If the user wanted to have full control over how input maps to output, then there is already python-evdev for that. The disadvantage of python-evdev is that it requires some boilerplate and it is difficult to write scripts that do not suffer from many different edge cases.

(Particularly, before I started on evsieve, I had about two dozen python scripts for different things, and regularly observed that writing a script that did thing A was relatively simple, writing a script that did thing B was relatively simple, but writing one that did both A and B was really difficult due to edge cases introduced by their interaction. Relatedly, the big time sink for adding new features to evsieve is not figuring out some way to implement it, but deciding on how that feature should interact with the other features that are already there, and figuring out what would be the most sensible behaviour for every edge case that could come up.)

Several projects have started to search for a higher-level way to describe ways to transform events. These higher-level configurations tend to make simple things easier but difficult things harder or impossible. The big question is how flexible you want your configuration language too be: if your configuration is too simple, many things users might want become impossible. If it is too complex, it ceases to offer much advantage over just writing a python-evdev script.

Many different projects have struck the balance between simplicity of configuration and versatiliy at different points. Before you start working on implementation details, I think it is important to first of all figure out exactly which kinds of transformations you intend to support and how you intend to present that in an user-friendly way to the user.

Since we can't beat python-evdev on versatiliy, we need to beat it on ease-of-use and user-friendliness. Having a user-friendly user interface for your targeted level of versatiliy is where the value of keymapping programs lies.

In particular,

Nice to have if feasable

• handle more complex input like key combinations

I think that whether, how, and which "complex input" you intend to support—along with how you intend to present that configurability to the user—is a fundamental question that needs to be considered before anything else, rather than treated as an afterthought. The answer to this question will impact just about every other part of the development process.

[sezanzeb]

In our case it would be the "mapping handler" architecture, which is like a pipeline, combined out of multiple handlers that can do different things. As far as we are know it is finished on beta. We'll have to wait and see if someone raises issues about certain things not being possible. It allows for example to combine mouse-movements with button clicks to produce some other output.

[kermitfrog]

@sezanzeb

although some polish is needed

You are very welcome to tweak it in Glade and to make a PR

Learning GTK / Glade is out of scope for me, but I'll create issues so you know what I mean.

Device > Preset > Mapping

and also to create a new issue to discuss this. Showing how the GUI would have to change, and explaining how the workflow of recording input would change would be helpful there :)

My plans for the UI are incomplete, but it will involve a TreeView to represent the hierarchy (group -> [subgroubs ..] -> window -> title) while mappings for all devices for that preset should be visible in the same view.
There is also the matter of "can be made to work" vs. "works well". If I were to transfer my current inputmangler configuration to input-remapper (using xautopresets) I'll probably end up with the configuration spread over more than a hundred files. Also in inputmangler, mappings can be inherited - I don't think that can be currently represented in input-remapper.
So the config format would have to change to make it work well.
I have to think a bit more about it..

Speaking of UI - I'm currently learning QML/Kirigami and have plenty of experience with the rest of Qt (mostly in C++ though), so I might help a bit with your Qt port. More by answering questions though, as it's not a high priority for me right now.

@KarsMulder

My two cents: I think that the hardest part of a keymapper project is actually not the implementation, but the design.

Yeah, I totally agree. One thing I'd like to explore here is which projects have (partially) compatible designs.

Since we can't beat python-evdev on versatiliy, we need to beat it on ease-of-use and user-friendliness.

Yep. But don't forget about performance. I don't like to have tools running in the background that use up more ressources than they need..

Do any of you have detailed documentation, describing your projects design?

[snyball]

If I were to create Hawck from scratch, this is the architecture I'd probably go with:

A single input-capture-redirect service with a small custom sandboxed VM, runs as a user with access to input, and accepts
literally any piece of code given to it on a UNIX socket from the desktop user. Safe because the VM cannot interact with the outside world, which was rarely needed in practice anyway. And similar to @kermitfrog I'd want to write this new version in Rust rather than C++.

The system should have access to not just keyboard/mouse/controller input but also many xdg-desktop-portal extensions, preferably the portable ones, and should include some wm-specific functionality that doesn't exist portably for Wayland compositors right now (like currently focused window.) Also random number generator, tty-detection, open-in-browser, etc.

As for launching things, I think we could provide functionality for launching .desktop files, but only from /usr/share (and never using the users $PATH or really any of their env) without thwarting the Wayland security model.

Then any GUI-based thing can just talk to this input service, and it should be flexible enough to do whatever one of those GUIs might want to do, and any text-based system can be compiled to the VMs bytecode.

I've been thinking about building this service just for fun, but it has ended up on the back-burner for a while because low-level Linux input stuff can be kinda frustrating due to a lack of documentation in a few areas.

If anyone else thinks this is a good idea, I'd write a spec for this architecture for reuse in other projects.

Of course, 99.9% of users are looking for one of a few select specific things like replacing caps-lock with ctrl/escape, but I still think a highly generic but safe and fast keyboard remapping system is a nice-to-have for the platform.

[jonasBoss]

I started work on the InputRemapper beta branch a year ago in order to solve my personal needs (using a 3DConnexion SpaceMouse as Joystick). Which somehow escalated into reinventing the whole architecture. That pretty much confirms the concerns raised by @KarsMulder:

I think that the hardest part of a keymapper project is actually not the implementation, but the design.

That said, I think the current approach can accomplish almost any reasonable remapping (mouse/joystick -> keyboard and mouse <-> joystick) with support for combinations in each case. + macro support to generate complex input-sequences (I think it is possible to make keyboard-> joystick/mouse mappings with macros).

There are some limitations:

• Inputs need to be processed as whole frames (EV_SYN - EV_SYN), not on a per event basis: All macro keys on Alienware keyboard are detected as the same (EV_MSC not supported) sezanzeb/input-remapper#457 this needs to be addressed, as there are multiple bug reports with this issue.
• Cross device combinations are not possible. This is actually not true We do support combinations across multiple /dev/input/eventXX devices just not across different physical devices as they run in different processes. So this limitation is currently by choice not necessarily by design.
• Macros have limitations Trying to emulate hitbox controller rules sezanzeb/input-remapper#465 (reply in thread) @snyball suggestion for a sandboxed process could be a good solution to support proper scripting in macros.
• Python is slow. InputRemapper has quiet some overhead. It was never properly profiled, so there might be potential to optimize it. But there is no arguing that Rust or C++ would be much faster.

In general I think it is quite possible to design a common Sevice which is simple to use for simple tasks e.g. remapping of n inputs to one output. But also provides a api for user scripts and more complex behavior. Implementing a good

dbus interface (change preset, update config, create event ...)

will make it possible to develop different GUIs or simple scripts which may or may not maintain their own configurations and translate them for the service.

[sezanzeb]

across different physical devices as they run in different processes

I sometimes wonder if this limitation can be avoided. Soon mappings will hold the information of their source device, so we could as well just record from all devices at once I guess. Idk.

Python is slow. InputRemapper has quiet some overhead. It was never properly profiled, so there might be potential to optimize it. But there is no arguing that Rust or C++ would be much faster.

For performance, if there really are no good optimizations possible, I'd not be very opposed of translating everything to a different language. It probably doesn't matter which one, because python is pretty much one of the slowest widespread languages. Translating the Tests could be a bit tricky sometimes, but they cover a lot of edge cases and past bug reports, so that would be really nice to be able to keep them. But anyway, if someone could do some profiling that would be great.

@snyball suggestion for a sandboxed process could be a good solution to support proper scripting in macros.

a small custom sandboxed VM

Also see sezanzeb/input-remapper#500. I thought lua doesn't require a vm to sandbox it, or does it?