Back to the future: sequential alignment of text representations

Johannes Bjerva, Wouter M. Kouw, Isabelle Augenstein

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

43 Downloads (Pure)

Abstract

Language evolves over time in many ways relevant to natural language processing tasks. For example, recent occurrences of tokens 'BERT' and 'ELMO' in publications refer to neural network architectures rather than persons. This type of temporal signal is typically overlooked, but is important if one aims to deploy a machine learning model over an extended period of time. In particular, language evolution causes data drift between time-steps in sequential decision-making tasks. Examples of such tasks include prediction of paper acceptance for yearly conferences (regular intervals) or author stance prediction for rumours on Twitter (irregular intervals). Inspired by successes in computer vision, we tackle data drift by sequentially aligning learned representations. We evaluate on three challenging tasks varying in terms of time-scales, linguistic units, and domains. These tasks show our method outperforming several strong baselines, including using all available data. We argue that, due to its low computational expense, sequential alignment is a practical solution to dealing with language evolution.
Original languageEnglish
Title of host publicationAAAI Conference on Artificial Intelligence
PublisherAssociation for the Advancement of Artificial Intelligence
Pages7440-7447
Number of pages8
ISBN (Print)978-1-57735-835-0
DOIs
Publication statusPublished - 11 Nov 2019
Event34th AAAI conference on Artificial Intelligence (AAAI2020) - Hilton new York Midtown, New York, United States
Duration: 7 Feb 202012 Feb 2020
Conference number: 34
https://aaai.org/Conferences/AAAI-20/
https://aaai.org/Conferences/AAAI-20/aaai20call/

Publication series

NameAAAI Conference on Artificial Intelligence
Number5
Volume34

Conference

Conference34th AAAI conference on Artificial Intelligence (AAAI2020)
Abbreviated titleAAAI2020
Country/TerritoryUnited States
CityNew York
Period7/02/2012/02/20
Internet address

Keywords

  • Natural language processing (NLP)
  • Deep learning
  • Domain adaptation
  • Subspace alignment

Fingerprint

Dive into the research topics of 'Back to the future: sequential alignment of text representations'. Together they form a unique fingerprint.

Cite this