How to measure uncertainty in uncertainty sampling for active learning

Vu-Linh Nguyen, Mohammad Hossein Shaker, Eyke Hüllermeier

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

Various strategies for active learning have been proposed in the machine learning literature. In uncertainty sampling, which is among the most popular approaches, the active learner sequentially queries the label of those instances for which its current prediction is maximally uncertain. The predictions as well as the measures used to quantify the degree of uncertainty, such as entropy, are traditionally of a probabilistic nature. Yet, alternative approaches to capturing uncertainty in machine learning, alongside with corresponding uncertainty measures, have been proposed in recent years. In particular, some of these measures seek to distinguish different sources and to separate different types of uncertainty, such as the reducible (epistemic) and the irreducible (aleatoric) part of the total uncertainty in a prediction. The goal of this paper is to elaborate on the usefulness of such measures for uncertainty sampling, and to compare their performance in active learning. To this end, we instantiate uncertainty sampling with different measures, analyze the properties of the sampling strategies thus obtained, and compare them in an experimental study.
Original languageEnglish
Number of pages28
JournalMachine Learning
VolumeXX
Issue numberXX
Publication statusAccepted/In press - 2021

Fingerprint Dive into the research topics of 'How to measure uncertainty in uncertainty sampling for active learning'. Together they form a unique fingerprint.

Cite this