Pre-Conference Talk by Dhanuja Tharith Wanniarachchi WANNIARACHCHIGE | MIMIC: AI and AR-enhanced Multi-Modal, Immersive, Relative Instruction Comprehension

Situated AI agents are rapidly evolving to support natural human interaction by combining language, vision, and gesture cues. We present MImIC (Multi-Modal, Immersive, Relative Instruction Comprehension), a framework that enhances instruction comprehension through RGB and LiDAR sensing, spatial reasoning, and transformer-based language understanding. Unlike conventional systems that rely on fully-specified commands, MImIC resolves relative references such as “a chair taller than that table,” enabling more intuitive and natural communication. We demonstrate these capabilities through AIRFurn, an AR-enhanced furniture shopping application, and evaluate its performance in both lab and real-world environments. Our findings show significantly faster task completion, higher accuracy, and greater user satisfaction compared to existing AR-based interfaces, highlighting MImIC’s potential to shape the next generation of immersive AI assistants.

This is a Pre-Conference talk for The ACM international joint conference on Pervasive and Ubiquitous Computing (UbiComp 2025).