PhD Dissertation Proposal by TIAN Zichen| Principled Configuration Optimization for Parameter-Efficient Fine-Tuning of Foundation Models

ABOUT THE TALK

Full fine-tuning of large pre-trained models incurs prohibitive computational cost and risks catastrophic forgetting. Parameter-efficient fine-tuning (PEFT) addresses this by updating only a small set of additional parameters while keeping the pre-trained model frozen. Low-Rank Adaptation (LoRA) has emerged as the de facto PEFT method for both large language models and vision transformers. However, in current practice, LoRA is configured uniformly: the same adaptation at every network position, and the same treatment of every task and every class. This thesis formulates LoRA configuration as a constrained resource allocation problem: a fixed parameter budget distributed over the structure of the model and the distribution of the data. We establish that the optimal allocation is non-uniform, and that it can be derived from properties of the model and the data rather than found by search. Configuration alone moves accuracy between 8.1% and 91.1% at a fixed parameter budget, and principled non-uniform allocation achieves state-of-the-art performance with 47% fewer parameters. Specifically, we address structural configuration in single-task adaptation (CVPR 2025), the training objective under data imbalance (NeurIPS 2024), and multi-task parameter sharing at scale (ICLR 2026). Building on these results, we propose a unified framework that jointly determines the complete allocation in multi-task settings, replacing configuration search with measurement followed by allocation.

ABOUT THE SPEAKER

Zichen TIAN is a PhD candidate in the School of Computing and Information Systems, Singapore Management University, advised by Prof. Qianru Sun. He holds an M.Sc. from Nanyang Technological University and a B.Eng. from Beijing University of Posts and Telecommunications. His research focuses on parameter-efficient adaptation of foundation models, systematically addressing robustness and scalability challenges. He has published three first-author papers at NeurIPS 2024, CVPR 2025 (Highlight), and ICLR 2026, and co-authored several papers at CVPR 2022, CVPR 2023, and IEEE TMM. He is a recipient of the Presidential Doctoral Fellowship and the Doctoral Dean's List (AY24-25, AY25-26), and has been serving as a reviewer for 10+ top-tier AI conferences including NeurIPS, CVPR, ICML, ICLR, IJCV and TNNLS.