External validation of the Advanced Alert Monitor (AAM) in a Dutch Hospital

Risk stratification is crucial in hospital settings for the early identification of high-risk patients and improving outcomes. Traditional approaches have relied on expert opinion-based early warning scores (EWS). However, the emergence of structured data in electronic medical records (EMR) has enabled the development of data-driven EWS. The advanced alert monitor (AAM) is a data-driven EWS validated to predict unanticipated ICU admissions or mortality within 12 hours. It has shown superior performance compared to traditional EWSs like the National Early Warning Score (NEWS). Prospective clinical use of the AAM in an American study has been demonstrated to reduce mortality, ICU admissions, and length of stay. This study aims to assess the generalizability of the AAM in a Dutch hospital setting and compare its performance with the NEWS. The primary objective is to evaluate the AAM's accuracy and discriminatory power in predicting adverse clinical events. Secondary objectives include comparing the AAM's performance with NEWS and investigating the impact of model retraining on its performance in the new center. Results show that the AAM and locally optimized AAM (LOAAM) outperform NEWS in terms of precision and sensitivity outside the original American population. However, differences in performance between AAM and LO-AAM are observed, with LO-AAM demonstrating improvement over AAM. Challenges in generalizability arise due to differences in patient populations, local procedures, and cultures across healthcare centers. Therefore, from the results we hypothesize that local optimization is necessary to address AAM-related challenges and locally optimize the AAM.