用历史行为记忆减少大模型强化学习中的重复错误。
The Past Is Not Past: Memory-Enhanced Dynamic Reward Shaping
- 通过存储中间表示识别常见错误模式
- 对高频错误集群施加更大惩罚，提升采样多样性
- 在多个数据集上显著提升性能，适合改进LLM推理
尽管大语言模型的强化学习已取得成功，但常见问题在于采样多样性下降，导致策略反复生成相似错误。传统熵正则化仅增强当前策略的随机性，无法显式抑制跨轮次的重复失败模式。本文提出MEDS框架，将历史行为信号融入奖励设计，通过存储和利用中间模型表示，捕捉过往轮次特征，并使用基于密度的聚类识别频繁出现的错误模式。被分配到更常见错误簇的轮次会受到更重惩罚，从而鼓励更广泛的探索并减少重复错误。在五个数据集和三种基础模型上，MEDS持续优于现有基线，平均性能提升最高达4.13分（pass@1）和4.37分（pass@128）。进一步分析显示，结合LLM标注与定量多样性指标，MEDS显著提升了采样过程中的行为多样性。
原文摘要 · Abstract (English)
Despite the success of reinforcement learning for large language models, a common failure mode is reduced sampling diversity, where the policy repeatedly generates similar erroneous behaviors. Classical entropy regularization encourages randomness under the current policy, but does not explicitly discourage recurrent failure patterns across rollouts. We propose MEDS, a Memory-Enhanced Dynamic reward Shaping framework that incorporates historical behavioral signals into reward design. By storing and leveraging intermediate model representations, we capture features of past rollouts and use density-based clustering to identify frequently recurring error patterns. Rollouts assigned to more prevalent error clusters are penalized more heavily, encouraging broader exploration while reducing repeated mistakes. Across five datasets and three base models, MEDS consistently improves average performance over existing baselines, achieving gains of up to 4.13 pass@1 points and 4.37 pass@128 points. Additional analyses using both LLM-based annotations and quantitative diversity metrics show that MEDS increases behavioral diversity during sampling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。