Deep reinforcement learning-based prosumer aggregation bidding strategy in a hierarchical local electricity market

Haoyang Zhang; J.K. (Koen) Kok; N.G. Paterakis

doi:10.1109/EEE-AM58328.2023.10395533

Deep reinforcement learning-based prosumer aggregation bidding strategy in a hierarchical local electricity market

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

2 Citations (Scopus)

219 Downloads (Pure)

Abstract

This paper investigates the application of deep reinforcement learning (DRL) algorithm for the decision-support of a prosumer aggregation in a hierarchical local electricity market (LEM) comprising a peer-to-peer (P2P) market and a corrective market. The agent first submits bids/asks to the P2P market where prosumer aggregations are able to trade electricity directly with each other. After that, the agent participates in the corrective market, where the market operator formulates the corrective market as an AC optimal power flow (OPF) problem to ensure the system is operated within its operational limits. A DRL algorithm, namely Twin Delayed Deep Deterministic Policy Gradient (TD3), is used to find the strategic bidding strategy. The algorithm is tested on a real medium-voltage distribution grid to evaluate the effectiveness of the strategic bidding method. The result of the case study demonstrates that the agent can derive trading strategies to obtain high profits based on the TD3 algorithm.

Original language	English
Title of host publication	2023 Asia Meeting on Environment and Electrical Engineering, EEE-AM 2023
Editors	Zbigniew Leonowicz, Erika Stracqualursi
Publisher	Institute of Electrical and Electronics Engineers
Number of pages	6
ISBN (Electronic)	979-8-3503-8106-1
DOIs	https://doi.org/10.1109/EEE-AM58328.2023.10395533
Publication status	Published - 25 Jan 2024
Event	1st Asia Meeting on Environment and Electrical Engineering - Hanoi, Viet Nam Duration: 13 Nov 2023 → 15 Nov 2023

Conference

Conference	1st Asia Meeting on Environment and Electrical Engineering
Abbreviated title	EEE-AM 2023
Country/Territory	Viet Nam
City	Hanoi
Period	13/11/23 → 15/11/23

Funding

This publication is part of the research program ‘MEGAMIND – Enabling distributed operation of energy infrastructures through Measuring, Gathering, Mining and Integrating grid-edge Data’, (partly) financed by the Dutch Research Council (NWO), through the Perspectief funding instrument under number P19-25.

Funders	Funder number
Nederlandse Organisatie voor Wetenschappelijk Onderzoek	P19-25

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 7 Affordable and Clean Energy

Keywords

AC OPF
Deep reinforcement learning
Peer-to-peer market
Strategic bidding

Access to Document

10.1109/EEE-AM58328.2023.10395533

pdfFinal published version, 710 KBLicence: TAVERNE

Cite this

Zhang, H., Kok, J. K., & Paterakis, N. G. (2024). Deep reinforcement learning-based prosumer aggregation bidding strategy in a hierarchical local electricity market. In Z. Leonowicz, & E. Stracqualursi (Eds.), 2023 Asia Meeting on Environment and Electrical Engineering, EEE-AM 2023 Article 10395533 Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/EEE-AM58328.2023.10395533

@inproceedings{888f916dfb714a4ab58147f35dce0009,

title = "Deep reinforcement learning-based prosumer aggregation bidding strategy in a hierarchical local electricity market",

abstract = "This paper investigates the application of deep reinforcement learning (DRL) algorithm for the decision-support of a prosumer aggregation in a hierarchical local electricity market (LEM) comprising a peer-to-peer (P2P) market and a corrective market. The agent first submits bids/asks to the P2P market where prosumer aggregations are able to trade electricity directly with each other. After that, the agent participates in the corrective market, where the market operator formulates the corrective market as an AC optimal power flow (OPF) problem to ensure the system is operated within its operational limits. A DRL algorithm, namely Twin Delayed Deep Deterministic Policy Gradient (TD3), is used to find the strategic bidding strategy. The algorithm is tested on a real medium-voltage distribution grid to evaluate the effectiveness of the strategic bidding method. The result of the case study demonstrates that the agent can derive trading strategies to obtain high profits based on the TD3 algorithm.",

keywords = "AC OPF, Deep reinforcement learning, Peer-to-peer market, Strategic bidding",

author = "Haoyang Zhang and Kok, \{J.K. (Koen)\} and N.G. Paterakis",

year = "2024",

month = jan,

day = "25",

doi = "10.1109/EEE-AM58328.2023.10395533",

language = "English",

editor = "Zbigniew Leonowicz and Erika Stracqualursi",

booktitle = "2023 Asia Meeting on Environment and Electrical Engineering, EEE-AM 2023",

publisher = "Institute of Electrical and Electronics Engineers",

address = "United States",

note = "1st Asia Meeting on Environment and Electrical Engineering, EEE-AM 2023 ; Conference date: 13-11-2023 Through 15-11-2023",

}

Zhang, H , Kok, JK & Paterakis, NG 2024, Deep reinforcement learning-based prosumer aggregation bidding strategy in a hierarchical local electricity market. in Z Leonowicz & E Stracqualursi (eds), 2023 Asia Meeting on Environment and Electrical Engineering, EEE-AM 2023., 10395533, Institute of Electrical and Electronics Engineers, 1st Asia Meeting on Environment and Electrical Engineering, Hanoi, Viet Nam, 13/11/23. https://doi.org/10.1109/EEE-AM58328.2023.10395533

Deep reinforcement learning-based prosumer aggregation bidding strategy in a hierarchical local electricity market. / Zhang, Haoyang ; Kok, J.K. (Koen); Paterakis, N.G.
2023 Asia Meeting on Environment and Electrical Engineering, EEE-AM 2023. ed. / Zbigniew Leonowicz; Erika Stracqualursi. Institute of Electrical and Electronics Engineers, 2024. 10395533.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Deep reinforcement learning-based prosumer aggregation bidding strategy in a hierarchical local electricity market

AU - Zhang, Haoyang

AU - Kok, J.K. (Koen)

AU - Paterakis, N.G.

PY - 2024/1/25

Y1 - 2024/1/25

N2 - This paper investigates the application of deep reinforcement learning (DRL) algorithm for the decision-support of a prosumer aggregation in a hierarchical local electricity market (LEM) comprising a peer-to-peer (P2P) market and a corrective market. The agent first submits bids/asks to the P2P market where prosumer aggregations are able to trade electricity directly with each other. After that, the agent participates in the corrective market, where the market operator formulates the corrective market as an AC optimal power flow (OPF) problem to ensure the system is operated within its operational limits. A DRL algorithm, namely Twin Delayed Deep Deterministic Policy Gradient (TD3), is used to find the strategic bidding strategy. The algorithm is tested on a real medium-voltage distribution grid to evaluate the effectiveness of the strategic bidding method. The result of the case study demonstrates that the agent can derive trading strategies to obtain high profits based on the TD3 algorithm.

AB - This paper investigates the application of deep reinforcement learning (DRL) algorithm for the decision-support of a prosumer aggregation in a hierarchical local electricity market (LEM) comprising a peer-to-peer (P2P) market and a corrective market. The agent first submits bids/asks to the P2P market where prosumer aggregations are able to trade electricity directly with each other. After that, the agent participates in the corrective market, where the market operator formulates the corrective market as an AC optimal power flow (OPF) problem to ensure the system is operated within its operational limits. A DRL algorithm, namely Twin Delayed Deep Deterministic Policy Gradient (TD3), is used to find the strategic bidding strategy. The algorithm is tested on a real medium-voltage distribution grid to evaluate the effectiveness of the strategic bidding method. The result of the case study demonstrates that the agent can derive trading strategies to obtain high profits based on the TD3 algorithm.

KW - AC OPF

KW - Deep reinforcement learning

KW - Peer-to-peer market

KW - Strategic bidding

UR - https://www.scopus.com/pages/publications/85184849077

U2 - 10.1109/EEE-AM58328.2023.10395533

DO - 10.1109/EEE-AM58328.2023.10395533

M3 - Conference contribution

BT - 2023 Asia Meeting on Environment and Electrical Engineering, EEE-AM 2023

A2 - Leonowicz, Zbigniew

A2 - Stracqualursi, Erika

PB - Institute of Electrical and Electronics Engineers

T2 - 1st Asia Meeting on Environment and Electrical Engineering

Y2 - 13 November 2023 through 15 November 2023

ER -

Deep reinforcement learning-based prosumer aggregation bidding strategy in a hierarchical local electricity market

Abstract

Conference

Funding

UN SDGs

Keywords

Access to Document

Other files and links

Fingerprint

Cite this