用真实数据训练环境模型，提升自动出价的策略效果。
Permutation Equivariant Model-based Offline Reinforcement Learning for Auto-bidding
- 基于真实数据学习环境模型，结合生成数据扩展状态空间覆盖。
- 在真实场景中优于现有自动出价方法，显著提升收益表现。
- 适合研究在线广告竞价与强化学习落地的工程师和研究人员。
自动出价中的强化学习已从依赖简单离线模拟器（模拟驱动强化学习出价，SRLB）转向使用固定真实数据集的离线强化学习（离线强化学习出价，ORLB）。然而，ORLB策略受限于数据集的状态空间覆盖，提升有限；而SRLB虽拓展了状态覆盖，但存在模拟器与现实之间的差距，可能误导策略。本文提出基于模型的强化学习出价（MRLB），从真实数据中学习环境模型，结合真实与模型生成的数据训练策略，扩大状态覆盖范围。为确保模型可靠性，提出：1）排列等变模型架构以增强泛化能力；2）一种鲁棒的离线Q学习方法，对模型误差进行悲观惩罚。二者构成排列等变的基于模型离线强化学习（PE-MORL）算法。真实世界实验表明，PE-MORL优于当前最先进的自动出价方法。
原文摘要 · Abstract (English)
Reinforcement learning (RL) for auto-bidding has shifted from using simplistic offline simulators (Simulation-based RL Bidding, SRLB) to offline RL on fixed real datasets (Offline RL Bidding, ORLB). However, ORLB policies are limited by the dataset's state space coverage, offering modest gains. While SRLB expands state coverage, its simulator-reality gap risks misleading policies. This paper introduces Model-based RL Bidding (MRLB), which learns an environment model from real data to bridge this gap. MRLB trains policies using both real and model-generated data, expanding state coverage beyond ORLB. To ensure model reliability, we propose: 1) A permutation equivariant model architecture for better generalization, and 2) A robust offline Q-learning method that pessimistically penalizes model errors. These form the Permutation Equivariant Model-based Offline RL (PE-MORL) algorithm. Real-world experiments show that PE-MORL outperforms state-of-the-art auto-bidding methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。