Preterm infants in a neonatal intensive care unit (NICU) require continuous monitoring as their life is at serious risk. In current patient monitoring based on vital signs, however, multiple alarms are generated for the same critical event, causing alarm fatigue of caregivers and stress in patient and parents. Moreover, detection of clinical deterioration with vitals crossing predefined boundaries can only be done in hindsight, whereas an early warning of such deterioration would be much more valuable. Finally, current monitoring involves a variety of obtrusive sensors and wiring, interfering with the babies’ well-being. This projects aims to bring patient monitoring beyond current state of the art by fusing the vitals and use video monitoring to reduce false alarms; employing data analytics to detect deterioration earlier; and using video techniques for robust motion detection and unobtrusive monitoring.