The fluid interaction between humans working in a team depends upon a shared understanding of the task, the environment, and the subtleties of language. A robot operating in this context must do the same. This talk will investigate how a notional robot can plan routes through an aerial image given a text command. The robot must be able to perceive the world as humans do so that its actions reflect the nuances of natural language and human perception. Traditional navigation systems for this task combine discrete perception, language processing, and planning blocks that are trained separately with different performance specifications. They communicate with restrictive interfaces to ease development, but this constrains the information one module can transfer to another, potentially limiting overall performance.
The speaker will present a technique that uses deep learning to transform a text command and a static aerial image into a trajectory, supervised with a form of inverse reinforcement learning called Max-Margin Planning (MMP). The MMP loss guides the network to imitate expert trajectories and implicitly learns the relationship between a command, an image, and a corresponding path solution. A single deep network contains all subtasks which are trained simultaneously under a single performance metric.
We study the comprehension abilities of the algorithm using an extensible synthetic benchmark derived from a dataset for Visual Question Answering. We also offer preliminary results on a semi-synthetic data set that uses real-world aerial imagery and artificial commands.
This talk will include a mock teaching session on inverse reinforcement learning.
About the Speaker
Arne Suppé received his Ph.D. in robotics from Carnegie Mellon University in 2022. Starting in 2001, he worked at the Robotics Institute on a wide range of autonomous vehicles and intelligent systems, including transit buses, robot arms, advanced driver assistance systems, and perception for robot navigation.
Applied machine learning and artificial intelligence are common themes in his work. He is interested in how deep learning and reinforcement learning can address multistep reasoning problems that address challenges faced by the government and industry.
He is a lecturer-track faculty candidate.
