arXiv:2508.07452cs.LGcs.AI2025-08被引 1

将表征与强化学习分层博弈,稳定训练并降低估值偏差。

Stackelberg Coupling of Online Representation Learning and Reinforcement Learning

  • Q网络为领导者慢更新,编码器为跟随者快适应,形成分层博弈。
  • 在多种DQN变体上测试,显著降低贝尔曼误差方差和估值偏差。
  • 适合追求训练稳定性和理论可解释性的强化学习研究者。

深度Q学习在单一网络中联合学习表征与价值,虽有协同优势,但表征需不断适应非平稳的价值目标,而价值估计又依赖这些变化的表征,导致训练不稳定。结合自举目标的高方差,使离策略方法中的价值估计产生偏差。本文提出分层耦合表征与强化学习(SCORER)框架,将表征学习与Q学习视为层级博弈中的两个策略主体:Q函数作为领导者,通过较少更新固定策略;感知网络(编码器)作为追随者,频繁调整以学习最小化贝尔曼误差方差的表征。这种分工使Q函数最小化均方贝尔曼误差(MSBE),感知网络最小化其方差,从而降低偏差。不对称更新机制实现稳定共适应,优于传统单体结构的同步更新。所提框架构成双层优化问题,由两时间尺度算法近似求解,生成非对称学习动态。大量实验在DQN及其变体上验证,性能提升源于算法洞察而非模型复杂度增加。

原文摘要 · Abstract (English)

Deep Q-learning jointly learns representations and values within monolithic networks, promising beneficial co-adaptation between features and value estimates. Although this architecture has attained substantial success, the coupling between representation and value learning creates instability as representations must constantly adapt to non-stationary value targets, while value estimates depend on these shifting representations. This is compounded by high variance in bootstrapped targets, which causes bias in value estimation in off-policy methods. We introduce Stackelberg Coupled Representation and Reinforcement Learning (SCORER), a framework for value-based RL that views representation and Q-learning as two strategic agents in a hierarchical game. SCORER models the Q-function as the leader, which commits to its strategy by updating less frequently, while the perception network (encoder) acts as the follower, adapting more frequently to learn representations that minimize Bellman error variance given the leader's committed strategy. Through this division of labor, the Q-function minimizes MSBE while perception minimizes its variance, thereby reducing bias accordingly, with asymmetric updates allowing stable co-adaptation, unlike simultaneous parameter updates in monolithic solutions. Our proposed SCORER framework leads to a bi-level optimization problem whose solution is approximated by a two-timescale algorithm that creates an asymmetric learning dynamic between the two players. Extensive experiments on DQN and its variants demonstrate that gains stem from algorithmic insight rather than model complexity.

强化学习分层博弈表征学习稳定性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。