Facebook develops new hand tracking software to type in VR without a keyboard
Facebook’s Reality Labs have announced today that their researchers are developing a hand tracking software that would allow touch typists to write in VR without the use of a physical keyboard.
It is a predictive motion model for marker-based hand tracking that could allow typists to match their physical typing speed by only tapping their fingers on a flat surface.
In the blog post announcing the new technology, researchers; Shangchen Han, Mark Richardson, and Robert Wang write,
“To support touch typing without a physical keyboard — and without the benefit of haptic feedback from individual keys — the team had to make sense of erratic typing patterns. They adopted statistical decoding techniques from automatic speech recognition, and where speech recognition uses an acoustic model to predict phonemes from audio frames, they instead use a motion model to predict keystrokes from hand motion. This, along with a language model, predicts what people intended to type despite ambiguous hand motion. Using this new method, typists averaged 73 words per minute with a 2.4% uncorrected error rate using their hands, a flat surface, and nothing else, achieving similar speed and accuracy to the same typist on a physical keyboard.
This surprising result led researchers to investigate why hand tracking was more effective than other physical methods, like tapping on a tablet. The team discovered hand tracking is uniquely good at isolating individual fingers and their trajectories as they reach for keys — information that is missing from capacitive sensing on tablets and smartphones today.”
According to Facebook, the software is only in the research stage, they currently have no plans to place it on any product roadmap. There are already Quest titles that make use of the simpler hand tracking software available; Waltz of the Wizard and The Line. Despite this promising research it looks like we will have to wait at least until the next generation of Oculus headsets to see this appear as a feature.