BRExIt: On Opponent Modelling in Expert Iteration

Daniel Hernandez, Hendrik Baier, Michael Kaisers

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

Abstract

Finding a best response policy is a central objective in game theory and multi-agent learning, with modern population-based training approaches employing reinforcement learning algorithms as best-response oracles to improve play against candidate opponents (typically previously learnt policies). We propose Best Response Expert Iteration (BRExIt), which accelerates learning in games by incorporating opponent models into the state-of-the-art learning algorithm Expert Iteration (ExIt). BRExIt aims to (1) improve feature shaping in the apprentice, with a policy head predicting opponent policies as an auxiliary task, and (2) bias opponent moves in planning towards the given or learnt opponent model, to generate apprentice targets that better approximate a best response. In an empirical ablation on BRExIt's algorithmic variants against a set of fixed test agents, we provide statistical evidence that BRExIt learns better performing policies than ExIt. Code available at: https://github.com/Danielhp95/on-opponent-modelling-in-expert-iteration-code. Supplementary material available at https://arxiv.org/abs/2206.00113.

Original languageEnglish
Title of host publicationProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23
EditorsEdith Elkind
PublisherInternational Joint Conferences on Artificial Intelligence (IJCAI)
Pages3795-3802
Number of pages8
ISBN (Electronic)978-195679203-4
Publication statusPublished - 2023
Event32nd International Joint Conference on Artificial Intelligence, IJCAI 2023 - Macao, China
Duration: 19 Aug 202325 Aug 2023

Conference

Conference32nd International Joint Conference on Artificial Intelligence, IJCAI 2023
Country/TerritoryChina
CityMacao
Period19/08/2325/08/23

Fingerprint

Dive into the research topics of 'BRExIt: On Opponent Modelling in Expert Iteration'. Together they form a unique fingerprint.

Cite this