PhD Dissertation Proposal by CHEN Zhi | Building Trustworthy AI for Code: From Model Training to Agent Evaluation and Benchmark Reliability

ABOUT THE TALK

AI for Code is evolving from code-completion models toward software development agents that inspect repositories, edit files, execute tests, and submit patches. As these systems participate in increasingly complex development workflows, final-output metrics alone cannot explain how results are produced, what risks they may hide, or whether the benchmarks used to measure them are reliable.

This dissertation studies trustworthy AI for Code through three connected perspectives: model training, agent evaluation, and benchmark reliability. It examines the effectiveness and memorization risks of collaborative code-model training; evaluates software agents through their generated patches, execution trajectories, testing behavior, and agent-written tests; and investigates the stability and validity of repository-level performance-optimization benchmarks. Across five studies, the dissertation shows that reported success should be interpreted beyond headline metrics. Effective training may still introduce memorization and leakage risks. Agents may pass tests while over-modifying code, following different solution paths from human developers, or failing to recover from execution errors. Agent-written tests may consume substantial resources without improving resolution outcomes, while optimization benchmark scores may be affected by runtime instability, reference patches, and scoring rules.

Together, these findings show that trustworthy AI for Code requires evaluating not only final correctness or performance, but also training risks, agent behavior, and the reliability of the underlying measurements.

ABOUT THE SPEAKER

Zhi CHEN is a Ph.D. student in Computer Science at Singapore Management University, supervised by Prof. Lingxiao Jiang. His research focuses on AI for Code and software development agents. He has published papers at leading software engineering conferences, including ICSE, ASE, and SANER. He also has industry R&D experience at technology companies such as TikTok AI Innovation Center and Sea Labs. More information is available at https://chenzhi-cz.github.io/.