arXiv:2605.20878cs.LG2026-05

用条件信息增益提升强化学习探索效率,兼顾长期与短期经验。

CIG: Exploration via Conditional Information Gain

论文配图:CIG: Exploration via Conditional Information Gain
图 1 · 摘自论文原文
  • 基于重放缓冲区和轨迹前缀的条件信息增益,实现可计算的探索奖励
  • 在12个任务上超越或媲美现有方法,对随机干扰保持鲁棒
  • 适用于高维状态空间,适合模型基于的强化学习场景

强化学习中的内在奖励机制通常只关注单一上下文:终身奖励依赖累积经验但忽略轨迹内冗余;周期性奖励惩罚轨迹内重复,却忽视长期进展。混合方法常通过启发式权重结合两者,或依赖无法扩展到高维状态空间的高斯过程动力学。轨迹级信息增益虽可分解为每步项,同时依赖重放缓冲区与滚动前缀,但对深度模型仍不可行。本文推导出可计算的条件信息增益(CIG)奖励:基于集成分歧核的对数行列式目标,其Cholesky分解生成因果性每步奖励,在保留双重条件的同时可扩展至高维状态空间。我们在模型基础上实例化CIG,其中滚动长度短,轨迹内修正尚未充分探索。在涵盖离散(MiniGrid)与连续控制(OGBench)的12个任务中,包括干净与带随机干扰的设定,CIG均优于或匹配现有探索方法,且对随机干扰具有鲁棒性。

原文摘要 · Abstract (English)

Intrinsic rewards for exploration in reinforcement learning condition on different contexts: lifelong rewards score each transition against accumulated experience but ignore within-rollout redundancy; episodic rewards penalize intra-trajectory repetition but discard lifetime progress. Hybrid methods combine both signals through heuristic weights or require Gaussian-process dynamics that do not scale beyond low-dimensional state spaces. Trajectory-level information gain decomposes into per-step terms that condition on the replay buffer and rollout prefix simultaneously, but remains intractable for deep models. We derive the Conditional Information Gain (CIG) reward as a tractable surrogate: a log-determinant objective over an ensemble disagreement kernel whose Cholesky factorization yields causal per-step rewards that retain both conditioning sets while scaling to high-dimensional state spaces. We instantiate CIG in a model-based setting, where rollouts are short and within-rollout corrections remain largely unexplored. Across twelve tasks spanning discrete (MiniGrid) and continuous control (OGBench), in both clean and stochastic-distractor settings, CIG outperforms or matches prior exploration methods while remaining robust to stochastic distractors.

强化学习探索策略信息增益高维状态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。