TongueBoard: An Oral Interface for Subtle Input

Abstract

We present TongueBoard, a retainer form-factor device for recognizing non-vocalized speech. TongueBoard enables absolute position tracking of the tongue by placing capacitive touch sensors on the roof of the mouth. We collect a dataset of 21 common words from four user study participants (two native American English speakers and two non-native speakers with severe hearing loss). We train a classifier that is able to recognize the words with 91.01% accuracy for the native speakers and 77.76% accuracy for the non-native speakers in a user dependent, offline setting. The native English speakers then participate in a user study involving operating a calculator application with 15 non-vocalized words and two tongue gestures at a desktop and with a mobile phone while walking. TongueBoard consistently maintains an information transfer rate of 3.78 bits per decision (number of choices = 17, accuracy = 97.1%) and 2.18 bits per second across stationary and mobile contexts, which is comparable to our control conditions of mouse (desktop) and touchpad (mobile) input.

Live Demo

As the first paper talk at Augmented Human 2019, we kicked off the conference with a live demo of our system! In this video captured by an audience member, Thad Starner is wearing a Tongueboard device, silently mouthing phrases that are recognized by our machine learning model and then "spoken" by my laptop's speaker. Hopefully, you can see some of the potential applications for silent speech recognition!

Download
Paper (PDF)
Reference

Richard Li, Jason Wu, and Thad Starner. 2019. TongueBoard: An Oral Interface for Subtle Input. In Proceedings of the 10th Augmented Human International Conference 2019 (AH2019). ACM, New York, NY, USA, Article 1, 9 pages. DOI: https://doi.org/10.1145/3311823.3311831