PhD Dissertation Proposal by YING Jiahao | Auto-Evaluation for Large Language Models

Auto-Evaluation for Large Language Models

ABOUT THE TALK

The rapid advancement of large language models (LLMs) has significantly expanded their potential applications across various domains. However, the development of reliable, scalable, and timely evaluation methodologies remains a critical challenge, as traditional manual evaluation approaches struggle to keep pace with the accelerating evolution of LLM capabilities. This dissertation addresses these challenges by proposing a comprehensive automatic evaluation (auto-evaluation) framework designed for large language models.

Guided by three key research questions, this dissertation systematically explores (1) methods for leveraging and extending existing evaluation datasets to better align with contemporary model capabilities, (2) the construction of automatic evaluators --- where LLMs themselves act as examiners, capable of generating, updating, and judging evaluation tasks in a reference-free manner, and (3) strategies for dynamic dataset expansion to mitigate issues such as benchmark leakage, data obsolescence, and unsustainable manual curation.

SPEAKER BIOGRAPHY

Jiahao Ying is a PhD candidate in Computer Science at the School of Computing and Information Systems at SMU, under the supervision of Assistant Professor Sun Qianru and external co-supervisor Cao Yixin. His interest is in LLMs evaluation.