Pyro: Thumb-Tip Gesture Recognition Using Pyroelectric Infrared Sensing


We present Pyro, a micro thumb-tip gesture recognition technique based on thermal infrared signals radiating from the fingers. Pyro uses a compact, low-power passive sensor, making it suitable for wearable and mobile applications. To demonstrate the feasibility of Pyro, we developed a self-contained prototype consisting of the infrared pyroelectric sensor, a custom sensing circuit, and software for signal processing and machine learning. A ten-participant user study yielded a 93.9% cross-validation accuracy and 84.9% leave-one-session-out accuracy on six thumb-tip gestures. Subsequent lab studies demonstrated Pyro’s robustness to varying light conditions, hand temperatures, and background motion. We conclude by discussing the insights we gained from this work and future research questions.

In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST’17)


Micro finger gestures offer new opportunities for natural, subtle, fast, and unobtrusive interactions in wearable, mobile, and ubiquitous computing applications. For example, gesturing the thumb tip against the tip of the index finger is a natural method of performing input, requiring little effort from users because the index finger serves as a supporting surface to naturally provide haptic feedback. This motion introduces less fatigue over time compared with traditional gestural input methods, which often require moving the finger, hand, or even the entire arm in mid-air. However, tracking fine-grained thumb-tip gestures remains very challenging due to the small magnitude of finger motions and frequent occurrences of self-occlusion.


We built our customized sensing board around a Cortex M4 micro-controller (MK20DX256VLH7) running at 96MHz, powered by the Teensy 3.2 firmware. The board has an LM324 based ADC preamp, a power management circuit, and a Bluetooth module. To reduce the dominant noise (50 kHz - 300 kHz) caused by powerline and fluorescent light ballasts, we implemented a bandpass filter with cut-off frequencies of 1.59 Hz and 486.75 Hz. The relatively wide bandwidth gives us the flexibility to explore sampling rates. After the noise is removed, the input signal is amplified with a gain of 33 and biased by AREF/2 (1.5 V) to preserve the fidelity of the analog signal. The gain value is carefully tuned to have an optimal sensing range of approximately 0.5 cm to 30 cm away from the PIR sensor. This design mitigates the background thermal infrared signals from the human body minimizing the impact on the foreground finger gesture signal.


Signal Processing

We collected raw gesturing data from ADC in microcontroller and extracted frequency domain and time domain features from them. We then fed the extracted features into RandomForest classifier getting the classification result. Here are typical samples of six thumb-tip micro gestures.

Six Thumb-tip Gestures Signal Processing

Selected Press Coverage

EurekAlert && ACM TechNews SIGCHI Edition (November 2017): Dartmouth to debut wearables that warn and wow at UIST 2017

The Dartmouth: Dartmouth team visits tech symposium