Research Seminar by CHEN Zhi and CAI Xuemeng

Talk #1: Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios
by CHEN Zhi, PhD Candidate

In recent years, AI-based software engineering has advanced from pre-trained models to agentic workflows, with Software Development Agents representing the next leap. These agents, capable of reasoning, planning, and interacting with environments, offer promising solutions to complex software tasks. However, while much research has evaluated code generated by large language models (LLMs), comprehensive studies on agent-generated patches in real-world settings are lacking. This study fills that gap by evaluating 4,892 patches from 10 top agents on 500 GitHub issues from SWE-Bench Verified, focusing on code quality impact. Our analysis shows no single agent dominated, with 170 issues unresolved, indicating room for improvement. Even patches that passed unit tests and resolved issues had different file and function modifications compared to gold patches from repository developers, revealing limitations in the benchmark’s test coverage. Most agents maintained code reliability and security, avoiding new bugs or vulnerabilities; while some increased code complexity, many reduced duplication and minimized code smells. Finally, agents performed better on simpler codebases, suggesting that breaking complex tasks into smaller sub-tasks could improve effectiveness. This study is the first comprehensive evaluation of agent-generated patches on real-world GitHub issues, offering insights to advance AI-driven software development.

Talk #2: Adapting Knowledge Prompt Tuning for Enhanced Automated Program Repair
by CAI Xuemeng, PhD Candidate

Automated Program Repair (APR) enhances software reliability by automatically generating bug-fixing patches. Recent work has advanced APR by fine-tuning pre-trained large language models (LLMs), such as CodeT5. However, fine-tuning loses effectiveness under data scarcity—a common issue. To address this, we adapt prompt tuning for APR and evaluate its performance using three LLMs of varying sizes and six datasets across four programming languages. Prompt tuning modifies a model’s input with extra prompt tokens and jointly tunes both the model and prompts on a small dataset. These tokens inject task-specific knowledge, which is critical when data is limited. Moreover, although domain knowledge is crucial in code intelligence, previous studies have not incorporated it into APR prompt tuning. To fill this gap, we introduce knowledge prompt tuning, integrating six types of code- or bug-related domain knowledge into APR. To our knowledge, this is the first work to evaluate prompt tuning—and domain knowledge incorporation—for APR under data scarcity. Our evaluation shows that prompt tuning with knowledge consistently outperforms fine-tuning across various settings, achieving an average improvement of 87.33% in limited data scenarios.

CHEN Zhi is a second-year Ph.D. student in Computer Science at Singapore Management University (SMU), under the supervision of Prof. Jiang Lingxiao. His research focuses on evaluating and exploring strategies to enhance automated software development.

Xuemeng CAI is a third-year PhD student and a Research Engineer at the Centre for Research on Intelligent Software Engineering (RISE) at Singapore Management University (SMU). She received her Bachelor’s degree in Computer Science and Design from the Singapore University of Technology and Design (SUTD) in 2022. Her research interests focus on software engineering challenges such as automatic program repair, code translation, and the interpretability of Large Language Models.