Hysteresis-Based RL: Robustifying Reinforcement Learning-based Control Policies via Hybrid Control

Jan De Priester; Ricardo G. Sanfelice; Nathan van de Wouw

doi:10.23919/ACC53348.2022.9867627

Hysteresis-Based RL: Robustifying Reinforcement Learning-based Control Policies via Hybrid Control

Jan De Priester
, Ricardo G. Sanfelice
, Nathan van de Wouw

Dynamics and Control

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

4 Citations (Scopus)

28 Downloads (Pure)

Abstract

Reinforcement learning (RL) is a promising approach for deriving control policies for complex systems. As we show in two control problems, the derived policies from using the Proximal Policy Optimization (PPO) and Deep Q-Network (DQN) algorithms may lack robustness guarantees. Motivated by these issues, we propose a new hybrid algorithm, which we call Hysteresis-Based RL (HyRL), augmenting an existing RL algorithm with hysteresis switching and two stages of learning. We illustrate its properties in two examples for which PPO and DQN fail.

Original language	English
Title of host publication	2022 American Control Conference, ACC 2022
Publisher	Institute of Electrical and Electronics Engineers
Pages	2663-2668
Number of pages	6
ISBN (Electronic)	9781665451963
DOIs	https://doi.org/10.23919/ACC53348.2022.9867627
Publication status	Published - 5 Sept 2022
Event	2022 American Control Conference, ACC 2022 - Atlanta, United States Duration: 8 Jun 2022 → 10 Jun 2022 https://acc2022.a2c2.org/

Conference

Conference	2022 American Control Conference, ACC 2022
Abbreviated title	ACC 2022
Country/Territory	United States
City	Atlanta
Period	8/06/22 → 10/06/22
Internet address	https://acc2022.a2c2.org/

Funding

*Research by R. G. Sanfelice has been partially supported by the National Science Foundation under Grant no. ECS-1710621, Grant no. CNS-1544396, and Grant no. CNS-2039054, by the Air Force Office of Scientific Research under Grant no. FA9550-19-1-0053, Grant no. FA9550-19-1-0169, and Grant no. FA9550-20-1-0238, and by the Army Research Office under Grant no. W911NF-20-1-0253. [email protected] 2Ricardo G. Sanfelice is with the Department of Electrical and Computer Engineering, University of California, Santa Cruz, CA 95064, USA; [email protected] 3Nathan van de wouw is with the Department of Mechanical Engineering, Eindhoven University of Technology, Eindhoven, 5612 AZ, Netherlands; [email protected]

Funders	Funder number
National Science Foundation	CNS-1544396, ECS-1710621, CNS-2039054
Air Force Office of Scientific Research (AFOSR)	FA9550-19-1-0169, FA9550-19-1-0053, FA9550-20-1-0238

Access to Document

10.23919/ACC53348.2022.9867627

pdfFinal published version, 464 KBLicence: TAVERNE

Cite this

@inproceedings{b385c9178ca64903a4d8e74793f896c5,

title = "Hysteresis-Based RL: Robustifying Reinforcement Learning-based Control Policies via Hybrid Control",

abstract = "Reinforcement learning (RL) is a promising approach for deriving control policies for complex systems. As we show in two control problems, the derived policies from using the Proximal Policy Optimization (PPO) and Deep Q-Network (DQN) algorithms may lack robustness guarantees. Motivated by these issues, we propose a new hybrid algorithm, which we call Hysteresis-Based RL (HyRL), augmenting an existing RL algorithm with hysteresis switching and two stages of learning. We illustrate its properties in two examples for which PPO and DQN fail.",

author = "\{De Priester\}, Jan and Sanfelice, \{Ricardo G.\} and \{van de Wouw\}, Nathan",

year = "2022",

month = sep,

day = "5",

doi = "10.23919/ACC53348.2022.9867627",

language = "English",

pages = "2663--2668",

booktitle = "2022 American Control Conference, ACC 2022",

publisher = "Institute of Electrical and Electronics Engineers",

address = "United States",

note = "2022 American Control Conference, ACC 2022, ACC 2022 ; Conference date: 08-06-2022 Through 10-06-2022",

url = "https://acc2022.a2c2.org/",

}

De Priester, J, Sanfelice, RG & van de Wouw, N 2022, Hysteresis-Based RL: Robustifying Reinforcement Learning-based Control Policies via Hybrid Control. in 2022 American Control Conference, ACC 2022., 9867627, Institute of Electrical and Electronics Engineers, pp. 2663-2668, 2022 American Control Conference, ACC 2022, Atlanta, Georgia, United States, 8/06/22. https://doi.org/10.23919/ACC53348.2022.9867627

Hysteresis-Based RL: Robustifying Reinforcement Learning-based Control Policies via Hybrid Control. / De Priester, Jan; Sanfelice, Ricardo G.; van de Wouw, Nathan.
2022 American Control Conference, ACC 2022. Institute of Electrical and Electronics Engineers, 2022. p. 2663-2668 9867627.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Hysteresis-Based RL

T2 - 2022 American Control Conference, ACC 2022

AU - De Priester, Jan

AU - Sanfelice, Ricardo G.

AU - van de Wouw, Nathan

PY - 2022/9/5

Y1 - 2022/9/5

N2 - Reinforcement learning (RL) is a promising approach for deriving control policies for complex systems. As we show in two control problems, the derived policies from using the Proximal Policy Optimization (PPO) and Deep Q-Network (DQN) algorithms may lack robustness guarantees. Motivated by these issues, we propose a new hybrid algorithm, which we call Hysteresis-Based RL (HyRL), augmenting an existing RL algorithm with hysteresis switching and two stages of learning. We illustrate its properties in two examples for which PPO and DQN fail.

AB - Reinforcement learning (RL) is a promising approach for deriving control policies for complex systems. As we show in two control problems, the derived policies from using the Proximal Policy Optimization (PPO) and Deep Q-Network (DQN) algorithms may lack robustness guarantees. Motivated by these issues, we propose a new hybrid algorithm, which we call Hysteresis-Based RL (HyRL), augmenting an existing RL algorithm with hysteresis switching and two stages of learning. We illustrate its properties in two examples for which PPO and DQN fail.

UR - https://www.scopus.com/pages/publications/85138494477

U2 - 10.23919/ACC53348.2022.9867627

DO - 10.23919/ACC53348.2022.9867627

M3 - Conference contribution

AN - SCOPUS:85138494477

SP - 2663

EP - 2668

BT - 2022 American Control Conference, ACC 2022

PB - Institute of Electrical and Electronics Engineers

Y2 - 8 June 2022 through 10 June 2022

ER -

Hysteresis-Based RL: Robustifying Reinforcement Learning-based Control Policies via Hybrid Control

Abstract

Conference

Funding

Access to Document

Other files and links

Fingerprint

Cite this