Pre-Conference Talk by CHENG Mingfei | Decictor: Towards Evaluating the Robustness of Decision-Making in Autonomous Driving Systems

Autonomous Driving System (ADS) testing is crucial in ADS development, with the current primary focus being on safety. However, the evaluation of non-safety-critical performance, particularly the ADS’s ability to make optimal decisions and produce optimal paths for autonomous vehicles (AVs), is also vital to ensure the intelligence and reduce risks of AVs. Currently, there is little work dedicated to assessing the robustness of ADSs’ path-planning decisions (PPDs), i.e., whether an ADS can maintain the optimal PPD after an insignificant change in the environment. The key challenges include the lack of clear oracles for assessing PPD optimality and the difficulty in searching for scenarios that lead to non-optimal PPDs. To fill this gap, we focus on evaluating the robustness of ADSs’ PPDs and propose the first method, Decictor, for generating nonoptimal decision scenarios (NoDSs), where the ADS does not plan optimal paths for AVs. Decictor comprises three main components: Non-invasive Mutation, Consistency Check, and Feedback. To overcome the oracle challenge, Non-invasive Mutation is devised to implement conservative modifications, ensuring the preservation of the original optimal path in the mutated scenarios. Subsequently, the Consistency Check is applied to determine the presence of nonoptimal PPDs by comparing the driving paths in the original and mutated scenarios. To deal with the challenge of large environment space, we design Feedback metrics that integrate spatial and temporal dimensions of the AV’s movement. These metrics can effectively steer the generation of NoDSs. We evaluate Decictor on Baidu Apollo, a production-grade ADS. The experimental results validate the effectiveness of Decictor in detecting non-optimal PPDs of ADSs.

This is a Pre-Conference talk for The 47th IEEE/ACM International Conference on Software Engineering (ICSE 2025).

CHENG Mingfei is a Ph.D. candidate in Computer Science at the SMU School of Computing and Information Systems, supervised by Assistant Professor XIE Xiaofei. His research mainly focuses on Testing Autonomous Driving Systems, and AI4SE.