Assessing the usefulness of Data Flow Diagrams for validating security threats

Context: Threat analysis is a pillar of security-by-design which plays an important role in the elicitation and refinement of security threats. In preparation for the analysis, a model of the system under analysis e.g., the Data Flow Diagram (DFD for short) is often created. Problem: Empirical measures of success are important for practitioners that are struggling to meet the current demands for expertise. But no previous work has investigated the role of these diagrams during the validation of identified security threats. Methods: This paper presents an experiment conducted with 98 students in two countries. We measured the impact of the DFD on the perceived and actual effectiveness of validating a list of identified security threats including both fabricated and actual threats. Results: In presence of sequence diagrams, the participants perceived DFDs as more useful. However, when exposed to both a DFD and a sequence diagram, DFDs had no significant impact on the participants’ ability to validate security threats.