Pre-Conference Talk by Ratnadira WIDYASARI | Demystifying Faulty Code: Step-by-Step Reasoning for Explainable Fault Localization

Manually pinpointing code elements that are associated with a fault is laborious and time-consuming. To overcome this challenge, various fault localization tools have been developed. These tools typically generate a ranked list of suspicious program elements. However, this information alone is insufficient. A prior study emphasized that automated fault localization should offer a rationale. In this study, we investigate the step-by-step reasoning for explainable fault localization. We explore the potential of Large Language Models (LLM) in assisting developers in reasoning about code. We proposed FuseFL that utilizes several combinations of information to enhance the LLM results which are spectrum-based fault localization results, test case execution outcomes, and code description (i.e., explanation of what the given code is intended to do). Our results demonstrate a 32.3% increase in the number of successfully localized faults at Top-1 compared to the baseline. To evaluate the explanations generated by FuseFL, we create a dataset of human explanations that provide step-by-step reasoning as to why specific lines of code are considered faulty. This dataset consists of 324 faulty code files, along with explanations for 600 faulty lines. From human studies that we conducted to evaluate the explanations, we found that  FuseFL generated correct explanations for 22 out of the 30 randomly sampled cases. 

This is a Pre-Conference talk for The IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER 2024).