Benchmarking the robustness of artificial intelligence tools for vestibular schwannoma and cochlea segmentation (crossMoDA 2021)

Introduction:
Fully automated artificial intelligence (AI) frameworks have recently reached outstanding performance for automatic segmentation of vestibular schwannomas (VS) from MRI. However, they often generalize poorly when the training data is acquired with scanners or imaging sequences that are different from those of the testing data. This problem strongly reduces the applicability of AI frameworks in real world radiosurgery settings. To increase the robustness of AI tools, various technical approaches have been proposed. However, these techniques have been validated either on private datasets or on small publicly available datasets. To tackle these limitations, the authors organized the CrossModality Domain Adaptation (crossMoDA) challenge in conjunction with the 24th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2021).
Methods:
The challenge’s goal was to segment two key brain structures involved in the follow-up and treatment planning of VS: the tumor and the cochlea. While contrast-enhanced T1 (ceT1) scans are commonly used for VS segmentation, recent work has demonstrated that high-resolution T2 (hrT2) imaging could be a reliable, safer, and lower-cost alternative to ceT1. For these reasons, the authors proposed an unsupervised cross-modality challenge to benchmark the generalization capability of techniques developed on images acquired with one sequence (ceT1) and tested on images acquired with another one (hrT2). Specifically, participants had access to a training set of unpaired annotated ceT1 (N=105) and non-annotated hrT2 (N=105). The automated segmentation tools developed by the challenge participants were then tested on a private hrT2 evaluation set (N=137). Images were collected on consecutive patients with a single sporadic VS treated with Gamma Knife stereotactic radiosurgery. Results:
A total of 341 teams registered for the challenge, allowing them to download the data. 55 teams from 16 countries submitted predictions to the validation leaderboard. Among them, 16 teams from 9 countries submitted their algorithm for the evaluation phase. The level of performance reached by the topperforming teams is strikingly high (best median Dice score - VS: 88.4%; Cochleas: 85.7%) and close to a state-of-the-art model developed using hrT2 scans and their corresponding annotations (median Dice score - VS: 92.5%; Cochleas: 87.7%).
Conclusions:
The authors organized the first international benchmark assessing the robustness of AI frameworks for stereotactic surgery planning of VS. The excellent results obtained by the top-performing teams suggest that AI tools can be robust to different imaging sequences. The next challenge edition will assess the robustness of AI tools for Koos grade classification.