DeepDIVE: Evaluating DeepCASE on Dataset Imbalance & Validity of Explanations

Abstract

A Security Operation Centre (SOCs) can be an important part of an organisation’s security strategy. Using Network Intrusion Detection Systems (NIDS), SOC analysts investigate alerts about suspicious activities detected in the network. Due to the large volume of alerts these NIDS may generate, an analyst can easily be overwhelmed. A technique to reduce the volume of alerts is DeepCASE. In this work, we evaluate the effect of label imbalance on DeepCASE’s performance, and show that the macro F1 score increases by as much as 69% when label imbalance is reduced. Furthermore, we evaluate the correctness of explanations that DeepCASE offers, and find that the correctness also improves when label imbalance is reduced. Tuning the detection rules used in SOCs, which can reduce the imbalance by 3 orders of magnitude, can therefore effectively improve classification performance and correctness of explanations offered by alert post-processing methods such as DeepCASE. We
assess other strategies for improving DeepCASE’s performance in the presence of label imbalance, such as focal loss and alert merging, and obtain no conclusive evidence to suggest that these methods can improve classification performance or explanation correctness.

Date of Award	2024
Original language	English
Supervisor	Luca Allodi (Supervisor 1)