储能交易中，融合价格、相对变化和预测信息的状态表示显著提升强化学习表现。
State Representation Matters in Deep Reinforcement Learning: Application to Energy Trading
- 用多种市场特征组合构建状态表示，对比其对策略性能的影响。
- 结合绝对价格、相对变化与预测信息后，跨区域平均收益达47.5%。
- 仅用单一特征类型效果不佳，组合设计对迁移能力至关重要。
能源交易决策不仅依赖当前市场价格，还涉及对未来市场状况的预期和运营约束。这使得强化学习智能体的状态表示成为关键设计选择。本文在水电蓄能套利环境HydroDam中，使用固定结构的Double DQN代理进行研究，保持环境、动作空间、奖励函数、网络结构与训练协议不变，仅改变市场特征输入。对比了绝对价格/日历特征、相对于近期市场历史的相对特征、预测特征，以及三类特征的所有组合。策略在2007–2011年比利时日前电价数据上训练，并在两个测试集上评估：2012–2025年的同一市场测试集和39个其他ENTSO-E市场区域。仅使用绝对特征时，测试集收益为28.8%，跨区域中位数仅为5.7%；仅用相对或预测特征也低于滚动价格基准。组合特征表现显著提升：绝对+相对达49.9%（测试集）和39.8%（跨区域中位数），绝对+相对+预测达55.6%和47.5%。结果表明，状态表示不是简单的预处理，而是策略设计的核心：稳健迁移需融合价格尺度、近期相对上下文与短期预测信息，而非依赖单一特征族。
原文摘要 · Abstract (English)
Energy trading decisions depend not only on current market prices, but also on expected future market conditions, and operational constraints. This makes the state representation given to a reinforcement learning agent an important design choice. We study this in HydroDam, a pumped-storage arbitrage environment, using a fixed Double DQN agent. The environment, action space, reward function, network, and training protocol are kept fixed; only the market features are changed. We compare absolute price/calendar features, relative features that compare current prices with recent market history, forecast features, and all combinations of these three feature families. Policies are trained and selected using 2007--2011 Belgian day-ahead prices and evaluated on two test settings: a later same-market test set from 2012--2025 and 39 other ENTSO-E market zones. Absolute features only reaches 28.8% on the test set and a median 5.7% across zones. Relative-only and forecast-only states also stay below a rolling price-score heuristic in the cross-zone median. Combining feature families is much stronger: absolute + relative reaches 49.9% on the test set and a 39.8% cross-zone median, while absolute + relative + forecast reaches 55.6% and 47.5%. These results suggest that state representation is not a minor preprocessing choice in storage-trading RL, but a central part of the policy design: robust transfer requires combining price scale, recent relative price context, and short-horizon forecast information, rather than relying on any single feature family.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。