PhD Dissertation Defense by PHAM Hong Quang | Continual Learning with Neural Networks

Empowering deep neural networks with the general learning capabilities to solve different problems sequentially has becoming an important step towards general intelligent machines. In this thesis, we study the problem of continual learning with deep neural networks and make two major contributions.

First, we study the classical continual learning framework and analyze how Batch Normalization affects different replay strategies. We discovered that although Batch Normalization facilitates continual learning, it also hinders the performance of older tasks. We named this the cross-task normalization phenomenon and conducted a comprehensive analysis to investigate and alleviate its negative effects.

Second, we develop the fast-and-slow learning framework as a novel approach to continual learning and beyond. The fast-and-slow learning idea motivates us to derive several novel continual algorithms that are effective in many challenging scenarios, including continual learning under domain shifts or limited training samples. Lastly, we extend the fast-and-slow learning framework beyond continual learning to address the challenging problem of online time series forecasting.

We hope this dissertation will be a promising step toward the next generation of intelligent machines with general and continual learning capabilities.

Quang Pham is currently a PhD candidate at School of Computing and Information Systems at Singapore Management University. He received his Bachelor degree from Vietnam National University, Ho Chi Minh City in 2015. His research interests include continual learning, machine learning, deep learning, and time series analysis.