Active learning with deep pre-trained models for sequence tagging of clinical and biomedical texts

Artem Shelmanov, Vadim Liventsev, Danil Kireev, Nikita Khromov, Alexander Panchenko, Irina Fedulova, Dmitry Dylov

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

7 Citations (Scopus)

Abstract

Active learning is a technique that helps to minimize the annotation budget required for the creation of a labeled dataset while maximizing the performance of a model trained on this dataset. It has been shown that active learning can be successfully applied to sequence tagging tasks of text processing in conjunction with deep learning models even when a limited amount of labeled data is available. Recent advances in transfer learning methods for natural language processing based on deep pre-trained models such as ELMo and BERT offer a much better ability to generalize on small annotated datasets compared to their shallow counterparts. The combination of deep pre-trained models and active learning leads to a powerful approach to dealing with annotation scarcity. In this work, we investigate the potential of this approach on clinical and biomedical data. The experimental evaluation shows that the combination of active learning and deep pre-trained models outperforms the standard methods of active learning. We also suggest a modification to a standard uncertainty sampling strategy and empirically show that it could be beneficial for annotation of very skewed datasets. Finally, we propose an annotation tool empowered with active learning and deep pre-trained models that could be used for entity annotation directly from Jupyter IDE.
Original languageEnglish
Title of host publicationProceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019
EditorsIllhoi Yoo, Jinbo Bi, Xiaohua Tony Hu
PublisherInstitute of Electrical and Electronics Engineers
Pages482-489
Number of pages8
ISBN (Electronic)9781728118673
DOIs
Publication statusPublished - 6 Feb 2020
Event2019 IEEE International Conference on Bioinformatics and Biomedicine - Hard Rock Hotel San Diego, San Diego, United States
Duration: 18 Nov 201921 Nov 2019
Conference number: https://ieeebibm.org/BIBM2019/

Conference

Conference2019 IEEE International Conference on Bioinformatics and Biomedicine
Abbreviated titleBIBM2019
Country/TerritoryUnited States
CitySan Diego
Period18/11/1921/11/19

Keywords

  • active learning
  • biomedical texts
  • clinical narrative
  • deep pre-trained models
  • sequence tagging

Fingerprint

Dive into the research topics of 'Active learning with deep pre-trained models for sequence tagging of clinical and biomedical texts'. Together they form a unique fingerprint.

Cite this