用梯度与动量的耦合关系衡量学习进度，更抗噪声且自动排序任务难易。
Measuring Learning Progress via Gradient-Momentum Coupling
- 基于优化过程中的梯度-动量耦合设计新信号
- 在噪声环境下仍能识别有效学习样本，提升探索效率
- 适合用于强化学习中的好奇心驱动探索，尤其抗观测噪声
在强化学习中，衡量学习进展对好奇心驱动探索至关重要，但常用预测误差等信号常无法区分有意义的学习模式与随机噪声。本文提出梯度-动量耦合（Gradient-Momentum Coupling, GMC），一种源自优化动态的信号，通过测量每个样本梯度与其前序梯度动量的每参数归一化绝对乘积，量化该样本梯度对当前学习的贡献度。利用动量对噪声和振荡的天然过滤能力，GMC 能识别出真正推动参数更新的样本。控制实验表明，该信号具备噪声鲁棒性并能实现自发课程学习，优先处理学习速度快的任务。在 MiniGrid 上的实验显示，将现有好奇心架构中的预测误差替换为 GMC，可显著提升对观测噪声的鲁棒性。
原文摘要 · Abstract (English)
Measuring learning progress is essential for curiosity-driven exploration in reinforcement learning, but widely used signals such as prediction error often fail to distinguish meaningful, learnable patterns from random noise. This paper proposes Gradient-Momentum Coupling (GMC), a signal derived from optimization dynamics that quantifies how useful each sample's gradient is for ongoing learning by measuring its per-parameter normalized absolute product with the momentum from previous gradients. By leveraging momentum's natural filtering of noise and oscillations, GMC identifies samples that contribute to ongoing parameter updates. Controlled experiments demonstrate noise robustness and emergent curriculum learning, with the signal prioritizing tasks by learning speed rather than difficulty. Experiments on MiniGrid suggest that replacing prediction error with GMC within existing curiosity-driven architectures can improve robustness to observation noise.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。