Active learning with deep pre-trained models for sequence tagging of clinical and biomedical texts

Vadim Liventsev, Artem Shelmanov, Danil Kireev, Nikita Khromov, Alexander Panchenko, Dmitry Dylov

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

Abstract

Active learning is a technique that helps to minimize the annotation budget required for the creation of a labeled dataset while maximizing the performance of a model trained on this dataset. It has been shown that active learning can be successfully applied to sequence tagging tasks of text processing in conjunction with deep learning models even when a limited amount of labeled data is available. Recent advances in transfer learning methods for natural language processing based on deep pre-trained models such as ELMo and BERT offer a much better ability to generalize on small annotated datasets compared to their shallow counterparts. The combination of deep pre-trained models and active learning leads to a powerful approach to dealing with annotation scarcity. In this work, we investigate the potential of this approach on clinical and biomedical data. The experimental evaluation shows that the combination of active learning and deep pre-trained models outperforms the standard methods of active learning. We also suggest a modification to a standard uncertainty sampling strategy and empirically show that it could be beneficial for annotation of very skewed datasets. Finally, we propose an annotation tool empowered with active learning and deep pre-trained models that could be used for entity annotation directly from Jupyter IDE.
Original languageEnglish
Title of host publication2019 IEEE International Conference on Bioinformatics and Biomedicine
Publication statusAccepted/In press - 1 Oct 2019
Event2019 IEEE International Conference on Bioinformatics and Biomedicine - Hard Rock Hotel San Diego, San Diego, United States
Duration: 18 Nov 201921 Nov 2019
Conference number: https://ieeebibm.org/BIBM2019/

Conference

Conference2019 IEEE International Conference on Bioinformatics and Biomedicine
Abbreviated titleBIBM2019
CountryUnited States
CitySan Diego
Period18/11/1921/11/19

    Fingerprint

Keywords

  • active learning
  • deep pre-trained models
  • sequence tagging
  • clinical narrative
  • biomedical texts

Cite this

Liventsev, V., Shelmanov, A., Kireev, D., Khromov, N., Panchenko, A., & Dylov, D. (Accepted/In press). Active learning with deep pre-trained models for sequence tagging of clinical and biomedical texts. In 2019 IEEE International Conference on Bioinformatics and Biomedicine [B645]