SCIS Seminar by Dr. Tarek Abdelzaher, Dr. Soujanya Poria, Dr. Lin Shao & Dr. Archan Misra

Advances in self-supervised AI revolutionized modern machine intelligence, but important challenges remain when applying these solutions in IoT contexts - specifically, on lower-end distributed embedded devices with multimodal specialized sensors, where ample training data are not readily available. The talk discusses challenges in offering self-supervised machine intelligence services to support distributed embedded sensing applications. The intersection of IoT applications, real-time requirements, distribution challenges, and self-supervised AI motivates several important research directions. For example, how to adapt self-supervised training pipelines to the embedded sensing domain? Can one develop foundation models for IoT that offer extended inference capabilities from multimodal time-series data? How to endow these models with an understanding of space (and spatial signal propagation) in order for them to reconstruct the state of the physical environment from multiple distributed sensor observations? How to overcome the challenge of data scarcity when it comes to training such AI models with specialized sensor data that are not as widely available as text and images? The talk addresses the above questions and presents initial empirical results on using the answers to train small foundation models for embedded sensor data.

With the recent success of foundation models, an important research direction is to develop a foundation model that can generalize across tasks, objects, and robot embodiments, enabling robots to adapt and operate effectively in unstructured environments.

In this talk, I will present two of our recent methods that address complementary challenges in this pursuit: high-level generative planning and low-level contact synthesis. First, I will introduce FLIP, a flow-centric generative planning framework. FLIP generates long-horizon task plans from an initial image and natural language instructions, thereby supporting general-purpose manipulation planning. By representing actions as 'flows,' FLIP not only enables planning across diverse objects, robots, and tasks, but also provides rich guidance for long-horizon video generation. Next, I will present our contact synthesis model, which formulates manipulation as a contact synthesis problem to address the significant variability across objects, robots, and tasks. The model takes as input point cloud data of objects and robot arms, object physical properties, target motion, and a mask of the manipulation region. It outputs contact points on the object along with the corresponding contact forces or post-contact motion trajectories, enabling robots to execute the required manipulation tasks. I will conclude by discussing future directions for advancing general-purpose robotic intelligence.

Vision-language-action (VLA) models are quickly becoming central to embodied AI, building on the breakthroughs of large language models and vision-language models. Their promise lies in something simple yet profound: the ability to follow natural language instructions and turn them into real-world actions. In this talk, I’ll walk through ten milestones that mark the progress and challenges ahead for VLA models—ranging from multimodality and reasoning to data, evaluation, generalization across robots, efficiency, whole-body coordination, safety, intelligent agents, and human collaboration. Each of these represents both a technical challenge and a stepping stone toward truly capable embodied systems. I’ll also highlight emerging trends that are shaping the future: spatial understanding, modeling world dynamics, post-training refinements, and synthetic data generation. Together, these directions point to a roadmap for accelerating VLA models toward real-world deployment and broader societal impact. My goal is to spark discussion on how we, as a community, can shape this exciting trajectory—and bring VLA models from promising prototypes to widely adopted, trustworthy, and useful embodied intelligence.

Advances in machine intelligence capabilities for perception, decision making and navigation will allow robots to be used, beyond traditional manufacturing assembly lines, as co-workers in environments such as homes, shopping malls and industrial sites. However, current embodied AI models are simply too large and complex to permit on-device execution on resource-constrained pervasive platforms. Through this talk, I shall introduce a few key research directions that aim to reduce the sensing and computational overheads for a couple of canonical embodied AI tasks, such as 2D/3D visual grounding of human instructions and robotic task planning in dynamic environments.