How to exploit weaknesses in biomedical challenge design and organization

Annika Reinke; Matthias Eisenmann; Sinan Onogur; Marko Stankovic; Patrick Scholz; Peter M. Full; Hrvoje Bogunovic; Bennett A. Landman; Oskar Maier; Bjoern H. Menze; Gregory C. Sharp; Korsuk Sirinukunwattana; Stefanie Speidel; Fons van der Sommen; Guoyan Zheng; Henning Müller; Michal Kozubek; Tal Arbel; Andrew P. Bradley; Pierre Jannin; Annette Kopp-Schneider; Lena Maier-hein

doi:10.1007/978-3-030-00937-3_45

How to exploit weaknesses in biomedical challenge design and organization

Annika Reinke, Matthias Eisenmann, Sinan Onogur, Marko Stankovic, Patrick Scholz, Peter M. Full, Hrvoje Bogunovic, Bennett A. Landman, Oskar Maier, Bjoern H. Menze, Gregory C. Sharp, Korsuk Sirinukunwattana, Stefanie Speidel, Fons van der Sommen, Guoyan Zheng, Henning Müller, Michal Kozubek, Tal Arbel, Andrew P. Bradley, Pierre JanninAnnette Kopp-Schneider, Lena Maier-hein

Video Coding & Architectures

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

10 Citations (Scopus)

Abstract

Since the first MICCAI grand challenge organized in 2007 in Brisbane, challenges have become an integral part of MICCAI conferences. In the meantime, challenge datasets have become widely recognized as international benchmarking datasets and thus have a great influence on the research community and individual careers. In this paper, we show several ways in which weaknesses related to current challenge design and organization can potentially be exploited. Our experimental analysis, based on MICCAI segmentation challenges organized in 2015, demonstrates that both challenge organizers and participants can potentially undertake measures to substantially tune rankings. To overcome these problems we present best practice recommendations for improving challenge design and organization.

Original language	English
Title of host publication	Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings
Editors	Alejandro F. Frangi, Gabor Fichtinger, Julia A. Schnabel, Carlos Alberola-López, Christos Davatzikos
Place of Publication	Berlin
Publisher	Springer
Chapter	45
Pages	388-395
Number of pages	8
ISBN (Electronic)	978-3-030-00937-3
ISBN (Print)	978-3-030-00936-6
DOIs	https://doi.org/10.1007/978-3-030-00937-3_45
Publication status	Published - 13 Sep 2018
Event	21st International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2018 - Granada, Spain Duration: 16 Sep 2018 → 20 Sep 2018

Publication series

Name	Lecture Notes in Computer Science
Volume	11073
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	21st International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2018
Country	Spain
City	Granada
Period	16/09/18 → 20/09/18

Access to Document

10.1007/978-3-030-00937-3_45

Link to publication in Scopus

Fingerprint Dive into the research topics of 'How to exploit weaknesses in biomedical challenge design and organization'. Together they form a unique fingerprint.

Cite this

Reinke, A., Eisenmann, M., Onogur, S., Stankovic, M., Scholz, P., Full, P. M., Bogunovic, H., Landman, B. A., Maier, O., Menze, B. H., Sharp, G. C., Sirinukunwattana, K., Speidel, S., van der Sommen, F., Zheng, G., Müller, H., Kozubek, M., Arbel, T., Bradley, A. P., ... Maier-hein, L. (2018). How to exploit weaknesses in biomedical challenge design and organization. In A. F. Frangi, G. Fichtinger, J. A. Schnabel, C. Alberola-López, & C. Davatzikos (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings (pp. 388-395). (Lecture Notes in Computer Science; Vol. 11073). Springer. https://doi.org/10.1007/978-3-030-00937-3_45

Reinke, Annika ; Eisenmann, Matthias ; Onogur, Sinan ; Stankovic, Marko ; Scholz, Patrick ; Full, Peter M. ; Bogunovic, Hrvoje ; Landman, Bennett A. ; Maier, Oskar ; Menze, Bjoern H. ; Sharp, Gregory C. ; Sirinukunwattana, Korsuk ; Speidel, Stefanie ; van der Sommen, Fons ; Zheng, Guoyan ; Müller, Henning ; Kozubek, Michal ; Arbel, Tal ; Bradley, Andrew P. ; Jannin, Pierre ; Kopp-Schneider, Annette ; Maier-hein, Lena. / How to exploit weaknesses in biomedical challenge design and organization. Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings. editor / Alejandro F. Frangi ; Gabor Fichtinger ; Julia A. Schnabel ; Carlos Alberola-López ; Christos Davatzikos. Berlin : Springer, 2018. pp. 388-395 (Lecture Notes in Computer Science).

@inproceedings{490e97900d194b85ba9c716ee405d2a7,

title = "How to exploit weaknesses in biomedical challenge design and organization",

abstract = "Since the first MICCAI grand challenge organized in 2007 in Brisbane, challenges have become an integral part of MICCAI conferences. In the meantime, challenge datasets have become widely recognized as international benchmarking datasets and thus have a great influence on the research community and individual careers. In this paper, we show several ways in which weaknesses related to current challenge design and organization can potentially be exploited. Our experimental analysis, based on MICCAI segmentation challenges organized in 2015, demonstrates that both challenge organizers and participants can potentially undertake measures to substantially tune rankings. To overcome these problems we present best practice recommendations for improving challenge design and organization.",

author = "Annika Reinke and Matthias Eisenmann and Sinan Onogur and Marko Stankovic and Patrick Scholz and Full, {Peter M.} and Hrvoje Bogunovic and Landman, {Bennett A.} and Oskar Maier and Menze, {Bjoern H.} and Sharp, {Gregory C.} and Korsuk Sirinukunwattana and Stefanie Speidel and {van der Sommen}, Fons and Guoyan Zheng and Henning M{\"u}ller and Michal Kozubek and Tal Arbel and Bradley, {Andrew P.} and Pierre Jannin and Annette Kopp-Schneider and Lena Maier-hein",

year = "2018",

month = sep,

day = "13",

doi = "10.1007/978-3-030-00937-3_45",

language = "English",

isbn = "978-3-030-00936-6",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "388--395",

editor = "Frangi, {Alejandro F.} and Gabor Fichtinger and Schnabel, {Julia A.} and Carlos Alberola-L{\'o}pez and Christos Davatzikos",

booktitle = "Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings",

address = "Germany",

note = "21st International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2018 ; Conference date: 16-09-2018 Through 20-09-2018",

}

Reinke, A, Eisenmann, M, Onogur, S, Stankovic, M, Scholz, P, Full, PM, Bogunovic, H, Landman, BA, Maier, O, Menze, BH, Sharp, GC, Sirinukunwattana, K, Speidel, S, van der Sommen, F, Zheng, G, Müller, H, Kozubek, M, Arbel, T, Bradley, AP, Jannin, P, Kopp-Schneider, A & Maier-hein, L 2018, How to exploit weaknesses in biomedical challenge design and organization. in AF Frangi, G Fichtinger, JA Schnabel, C Alberola-López & C Davatzikos (eds), Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings. Lecture Notes in Computer Science, vol. 11073, Springer, Berlin, pp. 388-395, 21st International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2018, Granada, Spain, 16/09/18. https://doi.org/10.1007/978-3-030-00937-3_45

How to exploit weaknesses in biomedical challenge design and organization. / Reinke, Annika; Eisenmann, Matthias; Onogur, Sinan; Stankovic, Marko; Scholz, Patrick; Full, Peter M.; Bogunovic, Hrvoje; Landman, Bennett A.; Maier, Oskar; Menze, Bjoern H.; Sharp, Gregory C.; Sirinukunwattana, Korsuk; Speidel, Stefanie; van der Sommen, Fons; Zheng, Guoyan; Müller, Henning; Kozubek, Michal; Arbel, Tal; Bradley, Andrew P.; Jannin, Pierre; Kopp-Schneider, Annette; Maier-hein, Lena.

Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings. ed. / Alejandro F. Frangi; Gabor Fichtinger; Julia A. Schnabel; Carlos Alberola-López; Christos Davatzikos. Berlin : Springer, 2018. p. 388-395 (Lecture Notes in Computer Science; Vol. 11073).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - How to exploit weaknesses in biomedical challenge design and organization

AU - Reinke, Annika

AU - Eisenmann, Matthias

AU - Onogur, Sinan

AU - Stankovic, Marko

AU - Scholz, Patrick

AU - Full, Peter M.

AU - Bogunovic, Hrvoje

AU - Landman, Bennett A.

AU - Maier, Oskar

AU - Menze, Bjoern H.

AU - Sharp, Gregory C.

AU - Sirinukunwattana, Korsuk

AU - Speidel, Stefanie

AU - van der Sommen, Fons

AU - Zheng, Guoyan

AU - Müller, Henning

AU - Kozubek, Michal

AU - Arbel, Tal

AU - Bradley, Andrew P.

AU - Jannin, Pierre

AU - Kopp-Schneider, Annette

AU - Maier-hein, Lena

PY - 2018/9/13

Y1 - 2018/9/13

N2 - Since the first MICCAI grand challenge organized in 2007 in Brisbane, challenges have become an integral part of MICCAI conferences. In the meantime, challenge datasets have become widely recognized as international benchmarking datasets and thus have a great influence on the research community and individual careers. In this paper, we show several ways in which weaknesses related to current challenge design and organization can potentially be exploited. Our experimental analysis, based on MICCAI segmentation challenges organized in 2015, demonstrates that both challenge organizers and participants can potentially undertake measures to substantially tune rankings. To overcome these problems we present best practice recommendations for improving challenge design and organization.

AB - Since the first MICCAI grand challenge organized in 2007 in Brisbane, challenges have become an integral part of MICCAI conferences. In the meantime, challenge datasets have become widely recognized as international benchmarking datasets and thus have a great influence on the research community and individual careers. In this paper, we show several ways in which weaknesses related to current challenge design and organization can potentially be exploited. Our experimental analysis, based on MICCAI segmentation challenges organized in 2015, demonstrates that both challenge organizers and participants can potentially undertake measures to substantially tune rankings. To overcome these problems we present best practice recommendations for improving challenge design and organization.

UR - http://www.scopus.com/inward/record.url?scp=85053834551&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-00937-3_45

DO - 10.1007/978-3-030-00937-3_45

M3 - Conference contribution

SN - 978-3-030-00936-6

T3 - Lecture Notes in Computer Science

SP - 388

EP - 395

BT - Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings

A2 - Frangi, Alejandro F.

A2 - Fichtinger, Gabor

A2 - Schnabel, Julia A.

A2 - Alberola-López, Carlos

A2 - Davatzikos, Christos

PB - Springer

CY - Berlin

T2 - 21st International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2018

Y2 - 16 September 2018 through 20 September 2018

ER -

Reinke A, Eisenmann M, Onogur S, Stankovic M, Scholz P, Full PM et al. How to exploit weaknesses in biomedical challenge design and organization. In Frangi AF, Fichtinger G, Schnabel JA, Alberola-López C, Davatzikos C, editors, Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 - 21st International Conference, 2018, Proceedings. Berlin: Springer. 2018. p. 388-395. (Lecture Notes in Computer Science). https://doi.org/10.1007/978-3-030-00937-3_45