自适应调节探索奖励，提升稀疏奖励下的学习效率
Adaptive Correlation-Weighted Intrinsic Rewards for Reinforcement Learning
- 用状态相关权重动态调整内在奖励，避免人工调参
- 在MiniGrid上样本效率提升，训练更稳定
- 轻量设计适合实际部署，计算开销极小
我们提出ACWI（自适应相关加权内在奖励），一种自适应的内在奖励缩放框架，用于在稀疏奖励强化学习中动态平衡内在与外在奖励，以改善探索效果。不同于依赖手动调参标量系数的传统方法（常导致任务间性能不稳定或次优），ACWI在线学习状态相关的缩放系数。具体而言，ACWI引入一个轻量级贝塔网络，通过编码器架构从智能体状态直接预测内在奖励权重。缩放机制基于相关性目标优化，促使加权后的内在奖励与未来外在回报的折扣和对齐。该设计实现任务自适应的探索激励，同时保持计算高效与训练稳定。我们在MiniGrid的一组稀疏奖励环境中评估了ACWI。实验结果表明，相较于固定内在奖励基线，ACWI在样本效率和学习稳定性上均有持续提升，且计算开销极低。
原文摘要 · Abstract (English)
We propose ACWI (Adaptive Correlation Weighted Intrinsic), an adaptive intrinsic reward scaling framework designed to dynamically balance intrinsic and extrinsic rewards for improved exploration in sparse reward reinforcement learning. Unlike conventional approaches that rely on manually tuned scalar coefficients, which often result in unstable or suboptimal performance across tasks, ACWI learns a state dependent scaling coefficient online. Specifically, ACWI introduces a lightweight Beta Network that predicts the intrinsic reward weight directly from the agent state through an encoder based architecture. The scaling mechanism is optimized using a correlation based objective that encourages alignment between the weighted intrinsic rewards and discounted future extrinsic returns. This formulation enables task adaptive exploration incentives while preserving computational efficiency and training stability. We evaluate ACWI on a suite of sparse reward environments in MiniGrid. Experimental results demonstrate that ACWI consistently improves sample efficiency and learning stability compared to fixed intrinsic reward baselines, achieving superior performance with minimal computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。