PhD Dissertation Proposal by PHAM Hong Quang | Continual Learning with Neural Networks

Recent years have witnessed the remarkable success of deep neural networks in solving cognitive problems, even outperforming humans on some simple tasks such as recognizing objects and playing Atari games. Continual learning is a natural step towards a more general intelligence system that can learn continuously to solve many problems. In this setting, the model is trained on a stream of data arising from changing input and output distributions associated with different tasks. Although continual learning is natural for humans, it turns out to be extremely challenging for neural networks because they lack the ability to quickly adapt to new information while retaining the obtained knowledge. This dissertation proposal is dedicated to developing novel methods to enable continual learning in deep neural networks via two major contributions. First, we develop a holistic framework that simultaneously addresses the knowledge transfer to both new and old tasks, allowing for a better trade-off between learning new tasks and retaining the learned knowledge. Second, we conduct a meta-analysis of Batch Normalization (BN), a crucial component in modern neural network architectures. We analyze the merits and drawbacks of BN in continual learning and propose a novel normalization layer that enjoys BN’s benefits while alleviating its negative effects.

PHAM Hong Quang is a PhD candidate advised by Professor Steven HOI in the School of Computing and Information Systems, Singapore Management University. His research focuses on developing methods to enable continual learning with deep neural networks.