This is a collection of hands-free input demonstrations that demonstrate how we can use computer vision to change the way that we interact with a computer. Modern computers have 2 predominant input methods:
- Keyboard and Mouse
- Touch Input
Both of these are tactile (touch-based) inputs. Our goal is to show that we can eliminate the need for tactile inputs through computer vision. Although eliminating the tactile input may seem undesirable, it may be a solution to ergonomics issues, and a further polished hands-free input system may eliminate the necessity of additional equipment when using technology.
To use the volume controls, run the "volume_control_better.py" script. Place the thumb and index (pointer) finger together to toggle volume control. To activate changes to the device volume, lower the middle finger of your hand. To set the volume, raise your middle finger again.
To use the brightness controls, it is the same controls as the volume, except that it will toggle the brightness instead. Note: There is no demonstration animation above, because the brightness of a screen is not screen-capturable. One will have to take our word on this that it works.
To use the mouse controls, the tip of the thumb will be what toggles the mouse. To left click, lower the index (pointer) finger like one would on a standard mouse. To right click, lower the middle finger like one would on a standard mouse. To scroll a page up, put together the three middle fingers (so keep up every finger that's not a thumb or a pinky), and lower it. To scroll a page down, put together all fingers that are not a thumb, and lower them.
The following portions of this project have different dependencies:
- Mediapipe ("pip install mediapipe")
- OpenCV ("pip install opencv-python")
- NumPy ("pip install numpy")
- pycaw ("pip install pycaw", Link to the creator's github here: https://github.com/AndreMiras/pycaw)
- screen_brightness_control ("pip install screen_brightness_control", Link to the creator's github here: https://github.com/Crozzers/screen_brightness_control)
- pynput ("pip install pynput", Link to the creator's github here: https://github.com/moses-palmer/pynput)
- tkinter (no need to install on Windows)
The limitations of this are the usual ones that plague computer vision tasks.
- Dependence on lighting conditions (good lighting is imperative to a clean input, daylight will lead to significantly better framerates than nighttime).
- Framerate dependent on computer specifications. If the user's computer specifications are not optimal, the FPS will be less than optimal (I get about 15-25 FPS on a relatively weak computer)
- GPUs will not offer any performance benefit (we are not training any machine learning model)
Please contact me at andrewjych@gmail.com, if you want to collaborate or suggest improvements to this project.