Active learning with deep pre-trained models for sequence tagging of clinical and biomedical texts

Vadim Liventsev, Artem Shelmanov, Danil Kireev, Nikita Khromov, Alexander Panchenko, Dmitry Dylov

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

Abstract

Active learning is a technique that helps to minimize the annotation budget required for the creation of a labeled dataset while maximizing the performance of a model trained on this dataset. It has been shown that active learning can be successfully applied to sequence tagging tasks of text processing in conjunction with deep learning models even when a limited amount of labeled data is available. Recent advances in transfer learning methods for natural language processing based on deep pre-trained models such as ELMo and BERT offer a much better ability to generalize on small annotated datasets compared to their shallow counterparts. The combination of deep pre-trained models and active learning leads to a powerful approach to dealing with annotation scarcity. In this work, we investigate the potential of this approach on clinical and biomedical data. The experimental evaluation shows that the combination of active learning and deep pre-trained models outperforms the standard methods of active learning. We also suggest a modification to a standard uncertainty sampling strategy and empirically show that it could be beneficial for annotation of very skewed datasets. Finally, we propose an annotation tool empowered with active learning and deep pre-trained models that could be used for entity annotation directly from Jupyter IDE.
Original languageEnglish
Title of host publication2019 IEEE International Conference on Bioinformatics and Biomedicine
Publication statusAccepted/In press - 1 Oct 2019
Event2019 IEEE International Conference on Bioinformatics and Biomedicine - Hard Rock Hotel San Diego, San Diego, United States
Duration: 18 Nov 201921 Nov 2019
Conference number: https://ieeebibm.org/BIBM2019/

Conference

Conference2019 IEEE International Conference on Bioinformatics and Biomedicine
Abbreviated titleBIBM2019
CountryUnited States
CitySan Diego
Period18/11/1921/11/19

Fingerprint

Text processing
Problem-Based Learning
Sampling
Processing
Uncertainty
Deep learning

Keywords

  • active learning
  • deep pre-trained models
  • sequence tagging
  • clinical narrative
  • biomedical texts

Cite this

Liventsev, V., Shelmanov, A., Kireev, D., Khromov, N., Panchenko, A., & Dylov, D. (Accepted/In press). Active learning with deep pre-trained models for sequence tagging of clinical and biomedical texts. In 2019 IEEE International Conference on Bioinformatics and Biomedicine [B645]
Liventsev, Vadim ; Shelmanov, Artem ; Kireev, Danil ; Khromov, Nikita ; Panchenko, Alexander ; Dylov, Dmitry. / Active learning with deep pre-trained models for sequence tagging of clinical and biomedical texts. 2019 IEEE International Conference on Bioinformatics and Biomedicine. 2019.
@inproceedings{2cd4757b018945278573d4c08ed03af2,
title = "Active learning with deep pre-trained models for sequence tagging of clinical and biomedical texts",
abstract = "Active learning is a technique that helps to minimize the annotation budget required for the creation of a labeled dataset while maximizing the performance of a model trained on this dataset. It has been shown that active learning can be successfully applied to sequence tagging tasks of text processing in conjunction with deep learning models even when a limited amount of labeled data is available. Recent advances in transfer learning methods for natural language processing based on deep pre-trained models such as ELMo and BERT offer a much better ability to generalize on small annotated datasets compared to their shallow counterparts. The combination of deep pre-trained models and active learning leads to a powerful approach to dealing with annotation scarcity. In this work, we investigate the potential of this approach on clinical and biomedical data. The experimental evaluation shows that the combination of active learning and deep pre-trained models outperforms the standard methods of active learning. We also suggest a modification to a standard uncertainty sampling strategy and empirically show that it could be beneficial for annotation of very skewed datasets. Finally, we propose an annotation tool empowered with active learning and deep pre-trained models that could be used for entity annotation directly from Jupyter IDE.",
keywords = "active learning, deep pre-trained models, sequence tagging, clinical narrative, biomedical texts",
author = "Vadim Liventsev and Artem Shelmanov and Danil Kireev and Nikita Khromov and Alexander Panchenko and Dmitry Dylov",
year = "2019",
month = "10",
day = "1",
language = "English",
booktitle = "2019 IEEE International Conference on Bioinformatics and Biomedicine",

}

Liventsev, V, Shelmanov, A, Kireev, D, Khromov, N, Panchenko, A & Dylov, D 2019, Active learning with deep pre-trained models for sequence tagging of clinical and biomedical texts. in 2019 IEEE International Conference on Bioinformatics and Biomedicine., B645, 2019 IEEE International Conference on Bioinformatics and Biomedicine, San Diego, United States, 18/11/19.

Active learning with deep pre-trained models for sequence tagging of clinical and biomedical texts. / Liventsev, Vadim; Shelmanov, Artem; Kireev, Danil; Khromov, Nikita; Panchenko, Alexander; Dylov, Dmitry.

2019 IEEE International Conference on Bioinformatics and Biomedicine. 2019. B645.

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

TY - GEN

T1 - Active learning with deep pre-trained models for sequence tagging of clinical and biomedical texts

AU - Liventsev, Vadim

AU - Shelmanov, Artem

AU - Kireev, Danil

AU - Khromov, Nikita

AU - Panchenko, Alexander

AU - Dylov, Dmitry

PY - 2019/10/1

Y1 - 2019/10/1

N2 - Active learning is a technique that helps to minimize the annotation budget required for the creation of a labeled dataset while maximizing the performance of a model trained on this dataset. It has been shown that active learning can be successfully applied to sequence tagging tasks of text processing in conjunction with deep learning models even when a limited amount of labeled data is available. Recent advances in transfer learning methods for natural language processing based on deep pre-trained models such as ELMo and BERT offer a much better ability to generalize on small annotated datasets compared to their shallow counterparts. The combination of deep pre-trained models and active learning leads to a powerful approach to dealing with annotation scarcity. In this work, we investigate the potential of this approach on clinical and biomedical data. The experimental evaluation shows that the combination of active learning and deep pre-trained models outperforms the standard methods of active learning. We also suggest a modification to a standard uncertainty sampling strategy and empirically show that it could be beneficial for annotation of very skewed datasets. Finally, we propose an annotation tool empowered with active learning and deep pre-trained models that could be used for entity annotation directly from Jupyter IDE.

AB - Active learning is a technique that helps to minimize the annotation budget required for the creation of a labeled dataset while maximizing the performance of a model trained on this dataset. It has been shown that active learning can be successfully applied to sequence tagging tasks of text processing in conjunction with deep learning models even when a limited amount of labeled data is available. Recent advances in transfer learning methods for natural language processing based on deep pre-trained models such as ELMo and BERT offer a much better ability to generalize on small annotated datasets compared to their shallow counterparts. The combination of deep pre-trained models and active learning leads to a powerful approach to dealing with annotation scarcity. In this work, we investigate the potential of this approach on clinical and biomedical data. The experimental evaluation shows that the combination of active learning and deep pre-trained models outperforms the standard methods of active learning. We also suggest a modification to a standard uncertainty sampling strategy and empirically show that it could be beneficial for annotation of very skewed datasets. Finally, we propose an annotation tool empowered with active learning and deep pre-trained models that could be used for entity annotation directly from Jupyter IDE.

KW - active learning

KW - deep pre-trained models

KW - sequence tagging

KW - clinical narrative

KW - biomedical texts

M3 - Conference contribution

BT - 2019 IEEE International Conference on Bioinformatics and Biomedicine

ER -

Liventsev V, Shelmanov A, Kireev D, Khromov N, Panchenko A, Dylov D. Active learning with deep pre-trained models for sequence tagging of clinical and biomedical texts. In 2019 IEEE International Conference on Bioinformatics and Biomedicine. 2019. B645