用条件信息增益提升强化学习探索效率,兼顾长期与短期经验。
CIG: Exploration via Conditional Information Gain

- 基于重放缓冲区和轨迹前缀的条件信息增益,实现可计算的探索奖励
- 在12个任务上超越或媲美现有方法,对随机干扰保持鲁棒
- 适用于高维状态空间,适合模型基于的强化学习场景
强化学习中的内在奖励机制通常只关注单一上下文:终身奖励依赖累积经验但忽略轨迹内冗余;周期性奖励惩罚轨迹内重复,却忽视长期进展。混合方法常通过启发式权重结合两者,或依赖无法扩展到高维状态空间的高斯过程动力学。轨迹级信息增益虽可分解为每步项,同时依赖重放缓冲区与滚动前缀,但对深度模型仍不可行。本文推导出可计算的条件信息增益(CIG)奖励:基于集成分歧核的对数行列式目标,其Cholesky分解生成因果性每步奖励,在保留双重条件的同时可扩展至高维状态空间。我们在模型基础上实例化CIG,其中滚动长度短,轨迹内修正尚未充分探索。在涵盖离散(MiniGrid)与连续控制(OGBench)的12个任务中,包括干净与带随机干扰的设定,CIG均优于或匹配现有探索方法,且对随机干扰具有鲁棒性。
原文摘要 · Abstract (English)
Intrinsic rewards for exploration in reinforcement learning condition on different contexts: lifelong rewards score each transition against accumulated experience but ignore within-rollout redundancy; episodic rewards penalize intra-trajectory repetition but discard lifetime progress. Hybrid methods combine both signals through heuristic weights or require Gaussian-process dynamics that do not scale beyond low-dimensional state spaces. Trajectory-level information gain decomposes into per-step terms that condition on the replay buffer and rollout prefix simultaneously, but remains intractable for deep models. We derive the Conditional Information Gain (CIG) reward as a tractable surrogate: a log-determinant objective over an ensemble disagreement kernel whose Cholesky factorization yields causal per-step rewards that retain both conditioning sets while scaling to high-dimensional state spaces. We instantiate CIG in a model-based setting, where rollouts are short and within-rollout corrections remain largely unexplored. Across twelve tasks spanning discrete (MiniGrid) and continuous control (OGBench), in both clean and stochastic-distractor settings, CIG outperforms or matches prior exploration methods while remaining robust to stochastic distractors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。