PhD Dissertation Defense by LUO Zilin | Towards Efficient Continual Learning: From Memory Optimization to Foundation Models

ABOUT THE TALK

Continual learning, also termed lifelong learning, enables machine learning models to incrementally acquire new knowledge while mitigating the degradation of previously learned information--a capability essential for adapting to dynamic, real-world data environments. This dissertation investigates the core challenges of continual learning and extends its application to enhancing training efficiency in the era of foundation models. The first part of this dissertation addresses the constraints of few-shot exemplar storage with a novel compression framework. While leveraging class activation maps to downsample non-discriminative pixels, we introduce an adaptive masking model, optimized through bilevel optimization, to store more exemplars efficiently. The second part focuses on the data dynamics of continual learning. We model continuous distribution shifts using bell-shaped curves to simulate realistic streams. We identify the challenges of non-stationarity and class imbalance in such evolving environments, and propose a rate-dependent coreset selector for adaptive memory selection to address them. The final part discusses how continual learning (CL) principles can be leveraged to optimize the training lifecycle of foundation models, which have huge capacity but are also susceptible to catastrophic forgetting and prohibitive retraining costs.

ABOUT THE SPEAKER

LUO Zilin is a Ph.D. candidate at Singapore Management University, supervised by SUN Qianru. He conducts research on artificial intelligence and machine learning. His research interest lies in continual learning. Before joining SMU, Zilin graduated with a Bachelor’s degree in Electronic Information Engineering from University of Science and Technology of China. He likes running, playing bass and doing sports in his leisure time.