You Were Always on My Mind: Introducing Chef’s Hat and COPPER for Personalized Reinforcement Learning

Pablo Barros (Corresponding author), Anne C. Bloem, Inge M. Hootsmans, Lena M. Opheij, Romain H.A. Toebosch, Emilia Barakova, Alessandra Sciutti

Research output: Contribution to journalArticleAcademicpeer-review

1 Citation (Scopus)
19 Downloads (Pure)

Abstract

Reinforcement learning simulation environments pose an important experimental test bed and facilitate data collection for developing AI-based robot applications. Most of them, however, focus on single-agent tasks, which limits their application to the development of social agents. This study proposes the Chef’s Hat simulation environment, which implements a multi-agent competitive card game that is a complete reproduction of the homonymous board game, designed to provoke competitive strategies in humans and emotional responses. The game was shown to be ideal for developing personalized reinforcement learning, in an online learning closed-loop scenario, as its state representation is extremely dynamic and directly related to each of the opponent’s actions. To adapt current reinforcement learning agents to this scenario, we also developed the COmPetitive Prioritized Experience Replay (COPPER) algorithm. With the help of COPPER and the Chef’s Hat simulation environment, we evaluated the following: (1) 12 experimental learning agents, trained via four different regimens (self-play, play against a naive baseline, PER, or COPPER) with three algorithms based on different state-of-the-art learning paradigms (PPO, DQN, and ACER), and two “dummy” baseline agents that take random actions, (2) the performance difference between COPPER and PER agents trained using the PPO algorithm and playing against different agents (PPO, DQN, and ACER) or all DQN agents, and (3) human performance when playing against two different collections of agents. Our experiments demonstrate that COPPER helps agents learn to adapt to different types of opponents, improving the performance when compared to off-line learning models. An additional contribution of the study is the formalization of the Chef’s Hat competitive game and the implementation of the Chef’s Hat Player Club, a collection of trained and assessed agents as an enabler for embedding human competitive strategies in social continual and competitive reinforcement learning.

Original languageEnglish
Article number669990
Number of pages13
JournalFrontiers in Robotics and AI
Volume8
DOIs
Publication statusPublished - 16 Jul 2021

Funding

AS is supported by a Starting Grant from the European Research Council (ERC) under the European Union’s Horizon 2020 Research and Innovation Programme. G.A. no. 804388, wHiSPER.

FundersFunder number
European Union's Horizon 2020 - Research and Innovation Framework Programme804388
H2020 European Research Council

    Keywords

    • continual reinforcement learning
    • human modeling through online games
    • opponent-aware competitive learning
    • reinforcement learning
    • simulation environment for reinforcement learning

    Fingerprint

    Dive into the research topics of 'You Were Always on My Mind: Introducing Chef’s Hat and COPPER for Personalized Reinforcement Learning'. Together they form a unique fingerprint.

    Cite this