Research Seminar by Dr. Shubham JAIN | Beyond Voice: Designing Novel Interfaces for Next-gen AI-Assistants

Voice has become the default way we interact with our devices. As computing moves onto the body, into earables, smart glasses, and other next-generation wearables, speech is the natural choice of interaction modality because it is fast, hands-free, and requires no learning curve. But voice-first interfaces come with several costs. They break down in noisy environments, disrupt the people around us, expose private conversations to anyone within earshot, and remain out of reach for users with speech impairments. These constraints are more challenging on wearables, where screens are small or absent and conventional input is limited. 

This talk presents a line of research on beyond-voice interactions, introducing interfaces that sense physical motion, for example jaw motion, facial and muscle vibrations, and hand gestures, instead of relying on audible speech alone. I will walk through the systems built around this idea: JawSense and MuteIt, which recognize unvoiced commands from jaw movement captured by a low-cost ear-worn IMU; Jawthenticate, which turns that same articulatory signal into a biometric for user authentication; Unvoiced, which pairs earable sensing with large language models to reconstruct fluent interaction from silent articulation; and AccessWear, which extends this philosophy to contactless gestures for blind users. 

I will close with our newest work, Beyond-Voice, which positions articulatory motion as a first-class sensing modality for next-generation AI assistants; not a replacement for speech, but a complement that enables parallel silent communication channels, multimodal emotional wellbeing monitoring, and real-time pronunciation coaching, deployable through glasses, earphones, and even pin-style AI assistant form factors. Together, these systems chart a design space for AI assistants that are private, inclusive, and always available, regardless of whether a user can, or wants to, speak out loud.