Automated evaluation of crowdsourced annotations in the cultural heritage domain

A. Nottamkandath, J. Oosterman, D. Ceolin, W. Fokkink

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

7 Citations (Scopus)
3 Downloads (Pure)

Abstract

Cultural heritage institutions are employing crowdsourcing techniques to enrich their collection. However, assessing the quality of crowdsourced annotations is a challenge for these institutions and manually evaluating all annotations is not feasible. We employ Support Vector Machines and feature set selectors to understand which annotator and annotation properties are relevant to the annotation quality. In addition we propose a trust model to build an annotator reputation using subjective logic and assess the relevance of both annotator and annotation properties on the reputation. We applied our models to the Steve.museum dataset and found that a subset of annotation properties can identify useful annotations with a precision of 90%. However, our studied annotator properties were less predictive.

Original languageEnglish
Title of host publicationProceedings of the 10th International Workshop on Uncertainty Reasoning for the Semantic Web (URSW 2014) co-located with the 13th International Semantic Web Conference (ISWC 2014)
PublisherCEUR-WS.org
Pages25-36
Number of pages12
Publication statusPublished - 2014
Externally publishedYes
Event10th International Workshop on Uncertainty Reasoning for the Semantic Web (URSW2014) - Riva del Garda, Italy
Duration: 19 Oct 201419 Oct 2014
Conference number: 10
http://ceur-ws.org/Vol-1259/
http://c4i.gmu.edu/ursw/2014/

Publication series

NameCEUR Workshop Proceedings
Volume1259
ISSN (Print)1613-0073

Conference

Conference10th International Workshop on Uncertainty Reasoning for the Semantic Web (URSW2014)
Abbreviated titleURSW2014
Country/TerritoryItaly
CityRiva del Garda
Period19/10/1419/10/14
Internet address

Fingerprint

Dive into the research topics of 'Automated evaluation of crowdsourced annotations in the cultural heritage domain'. Together they form a unique fingerprint.

Cite this