用信息论控制幻觉,让强化学习想象更真实可靠。
GIRL: Generative Imagination Reinforcement Learning via Information-Theoretic Hallucination Control
- 引入跨模态信号和自适应信任区域,约束生成轨迹偏离真实分布。
- 在多个任务上降低38%~61%的潜空间滚动偏差,提升长期回报。
- 适合长时序、稀疏奖励等复杂环境下的高效强化学习应用。
基于模型的强化学习(MBRL)通过在想象的轨迹中优化策略来提升样本效率,但长期规划会因模型误差累积导致想象轨迹偏离训练流形。我们提出GIRL(生成式想象强化学习),一种基于潜在世界模型的框架,包含两个关键组件:首先,利用冻结的DINOv2基础模型生成的跨模态锚定信号,将潜在转移先验约束到语义一致的嵌入空间,惩罚不一致或不合理的预测;其次,一个不确定性自适应的信任区域瓶颈,将KL正则化视为带约束优化问题的拉格朗日乘子,通过期望信息增益和相对性能损失信号,将想象漂移限制在学习到的区域内。我们基于性能差值引理与积分概率度量重新推导了价值差距界,该界在折扣因子趋近于1时仍具信息量,并连接目标与真实环境中的遗憾。在DeepMind Control、Adroit手部操作和Meta-World(含视觉干扰)三个基准套件上的实验表明,GIRL相比DreamerV3在各任务上减少38%至61%的潜空间滚动漂移,提升渐近回报,且在长时序任务中所需环境交互更少。GIRL在稀疏奖励和高接触场景下也优于TD-MPC2。轻量化先验变体降低了推理开销,提升了计算效率。
原文摘要 · Abstract (English)
Model-based reinforcement learning (MBRL) improves sample efficiency by optimizing policies inside imagined rollouts, but long-horizon planning degrades when model errors compound and imagined trajectories drift off the training manifold. We introduce GIRL (Generative Imagination Reinforcement Learning), a latent world-model framework that addresses this failure mode with two key components. First, a cross-modal grounding signal derived from a frozen foundation model (DINOv2) anchors the latent transition prior to a semantically consistent embedding space, penalizing inconsistent or implausible predictions. Second, an uncertainty-adaptive trust-region bottleneck interprets the KL regularizer as the Lagrange multiplier of a constrained optimization problem, restricting imagination drift within a learned region calibrated by Expected Information Gain and a Relative Performance Loss signal. We re-derive a value-gap bound using the Performance Difference Lemma and Integral Probability Metrics, yielding a bound that remains informative as the discount factor approaches one and connects the objective to real-environment regret. Experiments across three benchmark suites, including DeepMind Control, Adroit Hand Manipulation, and Meta-World with visual distractors, show that GIRL reduces latent rollout drift by 38 to 61 percent across tasks relative to DreamerV3, improves asymptotic return, and requires fewer environment interactions on long-horizon tasks. GIRL also outperforms TD-MPC2 on sparse-reward and high-contact settings under standard evaluation metrics. A distilled-prior variant reduces inference overhead and improves computational efficiency relative to the full model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。