Pre-Conference Talk by QIAO Hezhe | VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems

Large language model-driven multi-agent systems (LLM-MAS) excel at complex tasks, yet unreliable agents remain a key bottleneck to system-level reliability. Automatic failure attribution is therefore critical, but existing approaches, such as direct prediction of agent-error pairs and agent-first failure attribution, rely on local logs of agents and miss global failures that only manifest over full interaction trajectories, such as cross-step inconsistencies and inter-agent coordination errors. Moreover, directly predicting failures induces a large combinatorial search space, hindering fine-grained attribution. To address these challenges, we propose VerifyMAS, a hypothesis verification framework for agent failure attribution. Instead of directly predicting faulty agents and error types, VerifyMAS formulates and verifies failure hypotheses against full trajectories. This verification-based approach decomposes attribution into trajectory-level error validation and fine-grained agent localization, providing an error-first attribution approach that captures global failure patterns while substantially reducing the search space. We further introduce a hypothesis-based data construction strategy grounded in a structured error taxonomy and fine-tune a specialized LLM verifier model for trajectory-level failure verification and agent attribution. Experiments on Aegis-Bench and Who&When show that VerifyMAS consistently improves diverse backbone models, including open-source Qwen and API-based GPT models, outperforming prior methods without sacrificing inference efficiency for long multi-agent trajectories. 

This is a Pre-Conference talk for The Fortieth Annual Conference on Neural Information Processing Systems (NeurIPS 2026).

QIAO Hezhe is a fourth-year Ph.D. candidate in Computer Science at the School of Computing and Information Systems, under the supervision of Prof. Pang Guansong. Prior to that, he received his MS degree from the University of Chinese Academy Science, China. His research interests include  multi agents system reliability, agent post training, and generalist model for anomaly detection.  He has published multiple CCF A papers as  (co)-first author, including NeurIPS x4, KDD x2, ICML x1, IJCAI x1, TKDE x1. His research has attracted 1000+ citations and received multiple awards, e.g., SMU Dean List x2 and President’s Award of the Chinese Academy of Sciences. He is also the main organizer and presenter of the IJCAI 2025 tutorial titled ‘Deep Learning for Graph Anomaly Detection’. and AAAI 2026 tutorial titled ‘Toward Foundation Models for Detecting Abnormal Activities on Graphs’.