An Empirical Assessment of Repositioning of Static Analysis Alarms

Static analysis tools typically generate a large number of alarms that require manual inspection. In prior work, repositioning of alarms is proposed to (1) merge multiple similar alarms together and replace them by a fewer alarms, and (2) report alarms as close as possible to the causes for their generation. The premise is that the proposed merging and reporting of alarms will reduce the manual inspection effort. However, this premise has not been evaluated.

To evaluate the premise, this paper presents an empirical study on the proposed merging and reporting of static alarms. The study is conducted using static analysis alarms generated on C programs, where the alarms are representative of the merging Vs non-merging and repositioning Vs non-repositioning situations in real-life code. The study is performed as within-subjects via Qualtrics. Developers were asked to manually inspect and determine whether assertions added corresponding to the alarms hold. Additionally, two cognitive tests (mental rotation and operation span) are also conducted to determine relationship in performance.

The empirical evaluation results indicate that, in contrast to the expectations, the merging and repositioning of alarms (1) does not reduce manual inspection effort or does not improve accuracy of the inspection results, and (2) sometimes have a negative impact. A closer look at the results suggest that the study results are inconclusive and a more detailed study needs to be performed to evaluate the premise.