This paper describes a novel model for training an event detection system based on object tracking. We propose to model the training as a multiple instance learning problem, which allows us to train the classifier from annotated events despite temporal ambiguities. We apply this technique to realize a Perimeter Intrusion Detection (PID) algorithm and employ image-based features to distinguish real objects from moving vegetation and other distractions. An earlier developed tracking system is extended with the proposed technique to create an on-line PID-event detection system. Experiments with challenging videos show a reduction of the number of false positives by a factor 2–3 and improve the F1 detection performance from 0.15 to 0.28, when compared to a commercially available PID algorithm.