Surrogate Model-Based Reinforcement Learning for Bidding Strategies in Local Flexibility Markets

Reinforcement Learning (RL) has emerged as a promising tool for designing bidding strategies in electricity markets. Online RL learns policies by directly interacting with the environment, adapting in real time through trial-and-error feedback, while offline RL trains solely on a fixed, pre-collected dataset without any interaction with the environment. However, existing online RL methods assume detailed knowledge of the market-clearing model, including grid constraints, network parameters, and other participants' bids, which is unrealistic in a real-world setting. Additionally, the exploration process bears the risk of financial losses for market participants. Purely offline RL, on the other hand, suffers from its reliance on historical data that may be limited and non-diverse; as a result, it cannot gather new feedback to explore unobserved scenarios, potentially leading to sub-optimal results. To address these limitations, this paper proposes a surrogate model-based RL approach. Firstly, a machine learning surrogate model is constructed to approximate the local flexibility market (LFM) clearing process, using historical bids and corresponding market outcomes of the market participant. The surrogate model predicts the profit of the market participant associated with each bid in terms of price and quantity, thus avoiding the need for the true market model. A twin delayed deep deterministic policy gradient (TD3) RL bidding agent is then trained on this learned surrogate LFM model, enabling systematic exploration with reduced financial risk. Comparative simulations in the case study with three approaches—online RL, offline RL, and the proposed surrogate-based RL—reveal that online RL achieves the highest profit. Offline RL yields the lowest profit, constrained by limited historical data. The surrogatebased RL strategy outperforms the offline approach, offering a practical balance between profitability, and exploration risk.