Evaluation framework for network intrusion detection systems for in-vehicle CAN

Modern vehicles are complex safety critical cyber physical systems, that are connected to the outside world, with all security implications it brings. Different network intrusion detection systems (NIDSs) proposed for the CAN bus, the predominant type of in-vehicle network, to improve security are hard to compare due to disparate evaluation methods adopted. In this paper we provide the means to compare CAN NIDSs on equal footing and evaluate the ones detailed in the literature. Based on this we observe some limitation of existing approaches and why in the CAN setting it is intrinsically difficult to distinguish benign from malicious payload. We argue that 'meaning-aware' detection (a concept we define) which is challenging (but perhaps not impossible) to create for this setting.