Soon, your smartwatch could do much more than track your steps or display messages. Researchers at Cornell and KAIST in South Korea have found a way to turn an ordinary smartwatch into a high-tech controller with a system called “WatchHand.”

The system uses the watch’s built-in speaker and microphone to follow your hand movements as they happen. It sends out silent sound waves that reflect off your hand and return to the watch, letting the device know exactly what your fingers are doing. This works like micro-sonar for your wrist and doesn’t need any extra bulky equipment.

From Watch to Hand Tracker

Older hand-tracking devices are often too big and awkward to wear every day. WatchHand stands out because it uses technology already built into a regular Android watch. An AI program on the watch listens to the sound wave echoes to create a 3D map of your hand’s position.

“WatchHand substantially lowers the barriers to hand-pose tracking,” said Jiwan Kim, a doctoral student at KAIST and co-lead author. “If any device has a single speaker and microphone, our approach is applicable.”

Since all the processing is done on the watch itself, your data stays private. The researchers tested the technology with 40 people and found it could accurately track finger movements and wrist turns, even in noisy places. Still, it has some limits. It doesn’t work well if you’re walking, and for now, it only works on Android.

Typing In Thin Air

Researchers want to make using computers feel more natural. For example, you might tap your fingers to skip a song or move your hand to control a virtual mouse. This technology could also help people with limited mobility or those using VR headsets.

“In the future, with this kind of hand-tracking technology, we might be able to track our typing with just our smartwatch,” said co-lead author Chi-Jung Lee, a doctoral student at Cornell. “Our hands can act as an input device with computers.”

Cheng Zhang, an associate professor at Cornell, added, “With just a software update, we can potentially unlock entirely new capabilities on millions of existing devices.”