How to exploit weaknesses in biomedical challenge design and organization

Annika Reinke, Matthias Eisenmann, Sinan Onogur, Marko Stankovic, Patrick Scholz, Peter M. Full, Hrvoje Bogunovic, Bennett A. Landman, Oskar Maier, Bjoern H. Menze, Gregory C. Sharp, Korsuk Sirinukunwattana, Stefanie Speidel, Fons van der Sommen, Guoyan Zheng, Henning Müller, Michal Kozubek, Tal Arbel, Andrew P. Bradley, Pierre JanninAnnette Kopp-Schneider, Lena Maier-hein

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

3 Citations (Scopus)

Abstract

Since the first MICCAI grand challenge organized in 2007 in Brisbane, challenges have become an integral part of MICCAI conferences. In the meantime, challenge datasets have become widely recognized as international benchmarking datasets and thus have a great influence on the research community and individual careers. In this paper, we show several ways in which weaknesses related to current challenge design and organization can potentially be exploited. Our experimental analysis, based on MICCAI segmentation challenges organized in 2015, demonstrates that both challenge organizers and participants can potentially undertake measures to substantially tune rankings. To overcome these problems we present best practice recommendations for improving challenge design and organization.

Original languageEnglish
Title of host publicationMedical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings
EditorsAlejandro F. Frangi, Gabor Fichtinger, Julia A. Schnabel, Carlos Alberola-López, Christos Davatzikos
Place of PublicationBerlin
PublisherSpringer
Chapter45
Pages388-395
Number of pages8
ISBN (Electronic)978-3-030-00937-3
ISBN (Print)978-3-030-00936-6
DOIs
Publication statusPublished - 13 Sep 2018
Event21st International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2018 - Granada, Spain
Duration: 16 Sep 201820 Sep 2018

Publication series

NameLecture Notes in Computer Science
Volume11073
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference21st International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2018
CountrySpain
CityGranada
Period16/09/1820/09/18

Fingerprint

Benchmarking
Best Practice
Experimental Analysis
Recommendations
Ranking
Segmentation
Design
Demonstrate
Community
Influence

Cite this

Reinke, A., Eisenmann, M., Onogur, S., Stankovic, M., Scholz, P., Full, P. M., ... Maier-hein, L. (2018). How to exploit weaknesses in biomedical challenge design and organization. In A. F. Frangi, G. Fichtinger, J. A. Schnabel, C. Alberola-López, & C. Davatzikos (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings (pp. 388-395). (Lecture Notes in Computer Science; Vol. 11073). Berlin: Springer. https://doi.org/10.1007/978-3-030-00937-3_45
Reinke, Annika ; Eisenmann, Matthias ; Onogur, Sinan ; Stankovic, Marko ; Scholz, Patrick ; Full, Peter M. ; Bogunovic, Hrvoje ; Landman, Bennett A. ; Maier, Oskar ; Menze, Bjoern H. ; Sharp, Gregory C. ; Sirinukunwattana, Korsuk ; Speidel, Stefanie ; van der Sommen, Fons ; Zheng, Guoyan ; Müller, Henning ; Kozubek, Michal ; Arbel, Tal ; Bradley, Andrew P. ; Jannin, Pierre ; Kopp-Schneider, Annette ; Maier-hein, Lena. / How to exploit weaknesses in biomedical challenge design and organization. Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings. editor / Alejandro F. Frangi ; Gabor Fichtinger ; Julia A. Schnabel ; Carlos Alberola-López ; Christos Davatzikos. Berlin : Springer, 2018. pp. 388-395 (Lecture Notes in Computer Science).
@inproceedings{490e97900d194b85ba9c716ee405d2a7,
title = "How to exploit weaknesses in biomedical challenge design and organization",
abstract = "Since the first MICCAI grand challenge organized in 2007 in Brisbane, challenges have become an integral part of MICCAI conferences. In the meantime, challenge datasets have become widely recognized as international benchmarking datasets and thus have a great influence on the research community and individual careers. In this paper, we show several ways in which weaknesses related to current challenge design and organization can potentially be exploited. Our experimental analysis, based on MICCAI segmentation challenges organized in 2015, demonstrates that both challenge organizers and participants can potentially undertake measures to substantially tune rankings. To overcome these problems we present best practice recommendations for improving challenge design and organization.",
author = "Annika Reinke and Matthias Eisenmann and Sinan Onogur and Marko Stankovic and Patrick Scholz and Full, {Peter M.} and Hrvoje Bogunovic and Landman, {Bennett A.} and Oskar Maier and Menze, {Bjoern H.} and Sharp, {Gregory C.} and Korsuk Sirinukunwattana and Stefanie Speidel and {van der Sommen}, Fons and Guoyan Zheng and Henning M{\"u}ller and Michal Kozubek and Tal Arbel and Bradley, {Andrew P.} and Pierre Jannin and Annette Kopp-Schneider and Lena Maier-hein",
year = "2018",
month = "9",
day = "13",
doi = "10.1007/978-3-030-00937-3_45",
language = "English",
isbn = "978-3-030-00936-6",
series = "Lecture Notes in Computer Science",
publisher = "Springer",
pages = "388--395",
editor = "Frangi, {Alejandro F.} and Gabor Fichtinger and Schnabel, {Julia A.} and Carlos Alberola-L{\'o}pez and Christos Davatzikos",
booktitle = "Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings",
address = "Germany",

}

Reinke, A, Eisenmann, M, Onogur, S, Stankovic, M, Scholz, P, Full, PM, Bogunovic, H, Landman, BA, Maier, O, Menze, BH, Sharp, GC, Sirinukunwattana, K, Speidel, S, van der Sommen, F, Zheng, G, Müller, H, Kozubek, M, Arbel, T, Bradley, AP, Jannin, P, Kopp-Schneider, A & Maier-hein, L 2018, How to exploit weaknesses in biomedical challenge design and organization. in AF Frangi, G Fichtinger, JA Schnabel, C Alberola-López & C Davatzikos (eds), Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings. Lecture Notes in Computer Science, vol. 11073, Springer, Berlin, pp. 388-395, 21st International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2018, Granada, Spain, 16/09/18. https://doi.org/10.1007/978-3-030-00937-3_45

How to exploit weaknesses in biomedical challenge design and organization. / Reinke, Annika; Eisenmann, Matthias; Onogur, Sinan; Stankovic, Marko; Scholz, Patrick; Full, Peter M.; Bogunovic, Hrvoje; Landman, Bennett A.; Maier, Oskar; Menze, Bjoern H.; Sharp, Gregory C.; Sirinukunwattana, Korsuk; Speidel, Stefanie; van der Sommen, Fons; Zheng, Guoyan; Müller, Henning; Kozubek, Michal; Arbel, Tal; Bradley, Andrew P.; Jannin, Pierre; Kopp-Schneider, Annette; Maier-hein, Lena.

Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings. ed. / Alejandro F. Frangi; Gabor Fichtinger; Julia A. Schnabel; Carlos Alberola-López; Christos Davatzikos. Berlin : Springer, 2018. p. 388-395 (Lecture Notes in Computer Science; Vol. 11073).

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

TY - GEN

T1 - How to exploit weaknesses in biomedical challenge design and organization

AU - Reinke, Annika

AU - Eisenmann, Matthias

AU - Onogur, Sinan

AU - Stankovic, Marko

AU - Scholz, Patrick

AU - Full, Peter M.

AU - Bogunovic, Hrvoje

AU - Landman, Bennett A.

AU - Maier, Oskar

AU - Menze, Bjoern H.

AU - Sharp, Gregory C.

AU - Sirinukunwattana, Korsuk

AU - Speidel, Stefanie

AU - van der Sommen, Fons

AU - Zheng, Guoyan

AU - Müller, Henning

AU - Kozubek, Michal

AU - Arbel, Tal

AU - Bradley, Andrew P.

AU - Jannin, Pierre

AU - Kopp-Schneider, Annette

AU - Maier-hein, Lena

PY - 2018/9/13

Y1 - 2018/9/13

N2 - Since the first MICCAI grand challenge organized in 2007 in Brisbane, challenges have become an integral part of MICCAI conferences. In the meantime, challenge datasets have become widely recognized as international benchmarking datasets and thus have a great influence on the research community and individual careers. In this paper, we show several ways in which weaknesses related to current challenge design and organization can potentially be exploited. Our experimental analysis, based on MICCAI segmentation challenges organized in 2015, demonstrates that both challenge organizers and participants can potentially undertake measures to substantially tune rankings. To overcome these problems we present best practice recommendations for improving challenge design and organization.

AB - Since the first MICCAI grand challenge organized in 2007 in Brisbane, challenges have become an integral part of MICCAI conferences. In the meantime, challenge datasets have become widely recognized as international benchmarking datasets and thus have a great influence on the research community and individual careers. In this paper, we show several ways in which weaknesses related to current challenge design and organization can potentially be exploited. Our experimental analysis, based on MICCAI segmentation challenges organized in 2015, demonstrates that both challenge organizers and participants can potentially undertake measures to substantially tune rankings. To overcome these problems we present best practice recommendations for improving challenge design and organization.

UR - http://www.scopus.com/inward/record.url?scp=85053834551&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-00937-3_45

DO - 10.1007/978-3-030-00937-3_45

M3 - Conference contribution

SN - 978-3-030-00936-6

T3 - Lecture Notes in Computer Science

SP - 388

EP - 395

BT - Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings

A2 - Frangi, Alejandro F.

A2 - Fichtinger, Gabor

A2 - Schnabel, Julia A.

A2 - Alberola-López, Carlos

A2 - Davatzikos, Christos

PB - Springer

CY - Berlin

ER -

Reinke A, Eisenmann M, Onogur S, Stankovic M, Scholz P, Full PM et al. How to exploit weaknesses in biomedical challenge design and organization. In Frangi AF, Fichtinger G, Schnabel JA, Alberola-López C, Davatzikos C, editors, Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings. Berlin: Springer. 2018. p. 388-395. (Lecture Notes in Computer Science). https://doi.org/10.1007/978-3-030-00937-3_45