Validation of three late-onset sepsis prediction models in hospitalized infants

Late-onset sepsis (LOS) poses a relatively high risk of mortality and morbidity in preterm infants, attributed to their vulnerable immune systems and the complex environment of the neonatal intensive care unit (NICU). Numerous researchers have explored predictive models using non-invasive vital sign data, such as heart rate variability (HRV), respiration, and motion, to enable early detection of LOS in preterm infants in NICUs, yielding promising results. However, the scarcity of independent validation raises concerns regarding future clinical implementation. In this study, we collected 49 patients in our NICU, including 12 LOS patients and 37 non-LOS patients, throughout their entire hospitalization period between June 2022 and December 2022. Using this dataset, we assessed three prediction models: 1) an HRV-based HeRO model, 2) a multi-channel feature-based xgboost model (MC-xgb), and 3) a raw RR interval-based end-to-end deep neural network (RR-dnn). MC-xgb and RR-dnn were developed in our previous studies, while HeRO is commercially available and has already been implemented in several hospitals. We evaluated the prediction performance of these models using the area under the receiver operating characteristic curve (AUC) and metrics including sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) across various horizontal time windows. During our validation, we observed that RR-dnn outperformed the other models by achieving the highest AUC (84.6%) for predicting late-onset sepsis (LOS) within the next 3 hours. Although HeRO displayed the highest PPV (37.6%) over the entire hospitalization period, the overall PPV for all models in most prediction time windows remained suboptimal. This validation study highlights the need for further investigation and refinement of these models before considering their clinical implementation.