Abstract
Since the first MICCAI grand challenge organized in 2007 in Brisbane, challenges have become an integral part of MICCAI conferences. In the meantime, challenge datasets have become widely recognized as international benchmarking datasets and thus have a great influence on the research community and individual careers. In this paper, we show several ways in which weaknesses related to current challenge design and organization can potentially be exploited. Our experimental analysis, based on MICCAI segmentation challenges organized in 2015, demonstrates that both challenge organizers and participants can potentially undertake measures to substantially tune rankings. To overcome these problems we present best practice recommendations for improving challenge design and organization.
| Original language | English |
|---|---|
| Title of host publication | Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings |
| Editors | Alejandro F. Frangi, Gabor Fichtinger, Julia A. Schnabel, Carlos Alberola-López, Christos Davatzikos |
| Place of Publication | Berlin |
| Publisher | Springer |
| Chapter | 45 |
| Pages | 388-395 |
| Number of pages | 8 |
| ISBN (Electronic) | 978-3-030-00937-3 |
| ISBN (Print) | 978-3-030-00936-6 |
| DOIs | |
| Publication status | Published - 13 Sep 2018 |
| Event | 21st International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2018 - Granada, Spain Duration: 16 Sep 2018 → 20 Sep 2018 |
Publication series
| Name | Lecture Notes in Computer Science |
|---|---|
| Volume | 11073 |
| ISSN (Print) | 0302-9743 |
| ISSN (Electronic) | 1611-3349 |
Conference
| Conference | 21st International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2018 |
|---|---|
| Country | Spain |
| City | Granada |
| Period | 16/09/18 → 20/09/18 |
Fingerprint
Cite this
}
How to exploit weaknesses in biomedical challenge design and organization. / Reinke, Annika; Eisenmann, Matthias; Onogur, Sinan; Stankovic, Marko; Scholz, Patrick; Full, Peter M.; Bogunovic, Hrvoje; Landman, Bennett A.; Maier, Oskar; Menze, Bjoern H.; Sharp, Gregory C.; Sirinukunwattana, Korsuk; Speidel, Stefanie; van der Sommen, Fons; Zheng, Guoyan; Müller, Henning; Kozubek, Michal; Arbel, Tal; Bradley, Andrew P.; Jannin, Pierre; Kopp-Schneider, Annette; Maier-hein, Lena.
Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings. ed. / Alejandro F. Frangi; Gabor Fichtinger; Julia A. Schnabel; Carlos Alberola-López; Christos Davatzikos. Berlin : Springer, 2018. p. 388-395 (Lecture Notes in Computer Science; Vol. 11073).Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review
TY - GEN
T1 - How to exploit weaknesses in biomedical challenge design and organization
AU - Reinke, Annika
AU - Eisenmann, Matthias
AU - Onogur, Sinan
AU - Stankovic, Marko
AU - Scholz, Patrick
AU - Full, Peter M.
AU - Bogunovic, Hrvoje
AU - Landman, Bennett A.
AU - Maier, Oskar
AU - Menze, Bjoern H.
AU - Sharp, Gregory C.
AU - Sirinukunwattana, Korsuk
AU - Speidel, Stefanie
AU - van der Sommen, Fons
AU - Zheng, Guoyan
AU - Müller, Henning
AU - Kozubek, Michal
AU - Arbel, Tal
AU - Bradley, Andrew P.
AU - Jannin, Pierre
AU - Kopp-Schneider, Annette
AU - Maier-hein, Lena
PY - 2018/9/13
Y1 - 2018/9/13
N2 - Since the first MICCAI grand challenge organized in 2007 in Brisbane, challenges have become an integral part of MICCAI conferences. In the meantime, challenge datasets have become widely recognized as international benchmarking datasets and thus have a great influence on the research community and individual careers. In this paper, we show several ways in which weaknesses related to current challenge design and organization can potentially be exploited. Our experimental analysis, based on MICCAI segmentation challenges organized in 2015, demonstrates that both challenge organizers and participants can potentially undertake measures to substantially tune rankings. To overcome these problems we present best practice recommendations for improving challenge design and organization.
AB - Since the first MICCAI grand challenge organized in 2007 in Brisbane, challenges have become an integral part of MICCAI conferences. In the meantime, challenge datasets have become widely recognized as international benchmarking datasets and thus have a great influence on the research community and individual careers. In this paper, we show several ways in which weaknesses related to current challenge design and organization can potentially be exploited. Our experimental analysis, based on MICCAI segmentation challenges organized in 2015, demonstrates that both challenge organizers and participants can potentially undertake measures to substantially tune rankings. To overcome these problems we present best practice recommendations for improving challenge design and organization.
UR - http://www.scopus.com/inward/record.url?scp=85053834551&partnerID=8YFLogxK
U2 - 10.1007/978-3-030-00937-3_45
DO - 10.1007/978-3-030-00937-3_45
M3 - Conference contribution
SN - 978-3-030-00936-6
T3 - Lecture Notes in Computer Science
SP - 388
EP - 395
BT - Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings
A2 - Frangi, Alejandro F.
A2 - Fichtinger, Gabor
A2 - Schnabel, Julia A.
A2 - Alberola-López, Carlos
A2 - Davatzikos, Christos
PB - Springer
CY - Berlin
ER -