Pre-Conference Talk by CHEN Zhi | Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios

AI-driven software development has rapidly advanced with the emergence of software development agents that leverage large language models (LLMs) to tackle complex, repository-level software engineering tasks. These agents go beyond just generating final code; they engage in multi-step reasoning, use various tools for code modification and debugging, and interact with execution environments to diagnose and iteratively resolve issues. However, most existing evaluations focus primarily on static analyses of final code outputs, providing limited insights into the agents’ dynamic problem-solving processes. In this work, we conduct an in-depth empirical study on 3,977 solving-phase trajectories and 3,931 testing-phase logs from 8 top-ranked agents evaluated on 500 GitHub issues in the SWE-Bench benchmark. Our analysis identifies common execution errors, examines their impact on issue resolution, and highlights the challenges agents face during practical software development.

This is a Pre-Conference talk for IEEE/ACM International Conference on Software Engineering (ICSE 2026).

CHEN Zhi is a third-year Ph.D. candidate in Computer Science at Singapore Management University (SMU), under the supervision of Prof. Jiang Lingxiao. His research broadly focuses on AI-driven software engineering, with a particular interest in software development agents, their behavior in real-world settings, and their evaluation and improvement for practical software development tasks. Before and during his Ph.D., he has gained extensive industry  R&D experience at several technology companies, including TikTok AI Innovation Center and Sea Labs. More information is available at: https://chenzhi-cz.github.io/