Towards useful anomaly detection for back office networks

In this paper we present a protocol-aware anomaly detection framework specifically designed for back office networks together with a new automatic method for feature selection that allows to dramatically reduce the false positive rate (FPR) without compromising the detection rate (DR). The system monitors SMB and MS-RPC (the main protocols in back office networks) and takes into consideration specific features of SMB such as the presence of file paths, which are noisy, yet contain information necessary to detect some attacks. As a part of the framework we introduce a new method to cut the FPR by carefully building and selecting the right set of features to be monitored. In back office networks this is a challenging task where manual selection requires carefully exploring the network traffic to choose from numerous potential features. Also features need to be resilient to irregularities in the traffic caused by human involvement. Our framework automates selection utilizing two new metrics to determine the ‘quality’ of a feature: stability, i.e. its robustness to false alarms and granularity, i.e. the relative amount of information contained. Our experiments show a significant improvement in FPR-DR trade-off when our framework is used to select features in detection of network-based exploits and malicious file accesses.