Crowd disagreement of medical images is informative

Research output: Contribution to journalArticle

3 Citations (Scopus)
37 Downloads (Pure)


Classifiers for medical image analysis are often trained with a single consensus label, based on combining labels given by experts or crowds. However, disagreement between annotators may be informative, and thus removing it may not be the best strategy. As a proof of concept, we predict whether a skin lesion from the ISIC 2017 dataset is a melanoma or not, based on crowd annotations of visual characteristics of that lesion. We compare using the mean annotations, illustrating consensus, to standard deviations and other distribution moments, illustrating disagreement. We show that the mean annotations perform best, but that the disagreement measures are still informative. We also make the crowd annotations used in this paper available at

Original languageEnglish
Article number1806.08174
Number of pages8
Publication statusPublished - 21 Jun 2018

Bibliographical note

Submitted to MICCAI LABELS 2018


  • cs.CV


Dive into the research topics of 'Crowd disagreement of medical images is informative'. Together they form a unique fingerprint.

Cite this