PhD Dissertation Proposal by LE Hung | Deep Learning for Video-grounded Dialogues

In recent years, we have witnessed significant progress in building systems with artificial intelligence. However, despite advancements in machine learning and deep learning, we are still far from achieving a perfect autonomous agent that can perceive the multi-dimensional information from the surrounding world and converse with humans in natural language. Towards this goal, this thesis proposal is dedicated to building intelligent systems in the task of video-grounded dialogue. Specifically, in a video-grounded dialogue, a system is required to hold a multi-turn conversation with humans about the content of a video. Given an input video, a dialogue history, and a question about the video, a system must understand the contextual information of dialogue, extract relevant information from the video, and construct a dialogue response that is both contextually relevant and video-grounded. Towards building such systems, this dissertation proposal aims to address three major challenges: (1) visual reasoning in spatio-temporal space, (2) language reasoning in multiple turns, and (3) natural language generation. To tackle these challenges, we explore several methods, including bidirectional reasoning in spatio-temporal space, contextual reasoning paths in dialogue turns, counterfactual data augmentation, and diagnostic benchmark development, among others.

Hung Le is a third-year Ph.D. candidate in the School of Computing and Information Systems, Singapore Management University, advised by Professor Steven C.H. Hoi and Dr. Nancy F. Chen (A*STAR). His research focuses on deep learning techniques for conversational AI, including the research of machine learning tasks such as video-grounded dialogues and task-oriented dialogues.