arXiv:2605.18592cs.LGcs.AI2026-05被引 1

用长期训练记忆优化评分标准,让大模型强化学习更稳定高效。

AMARIS: A Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learning

  • 引入持久化记忆存储训练过程中的评估数据,实现跨周期的评分标准迭代
  • 在多个任务上提升2.8分(GPQA-Diamond)和2.2分(IFBench),超越静态与局部更新基线
  • 适合需要持续优化评分体系的RL微调场景,尤其适合教育、医疗等高可靠性领域

基于评分标准的奖励塑造为大模型强化学习提供了可解释且可编辑的奖励信号,但现有自适应评分方法通常仅依据当前批次或实例级比较进行更新。这种局部视角忽略了训练过程中产生的诊断信息,导致难以追踪重复失败、评估历史修改效果或在早期标准饱和后提升要求。本文提出AMARIS——一种基于记忆增强的评分标准改进系统,将回放分析、步骤级摘要和评分更新记录存入持久化评估记忆中,并检索近期及语义相关的历史信息以修订评分标准。在科学、医学、指令遵循和创意写作任务上,分别在全局与实例特定评分设置下评估了AMARIS。结果表明,其优于静态、局部自适应及无记忆对照组,在GPQA-Diamond上提升2.8分,IFBench上提升2.2分;分析显示,记忆机制减少了评分标准的振荡,支持从初期错误修正到后期课程进阶的演进路径。AMARIS异步运行于常规强化学习循环之外,相比同步更新显著降低阻塞延迟。

原文摘要 · Abstract (English)

Rubric-based reward shaping provides interpretable and editable reward signals for fine-tuning LLMs via reinforcement learning (RL), but existing adaptive rubric methods typically update criteria from local evidence such as the current batch or instance-level comparisons. This local view discards diagnostic information produced during training, making it difficult to track recurring failures, evaluate previous rubric edits, or raise standards once earlier criteria become saturated. We introduce AMARIS, A Memory-Augmented Rubric Improvement System that grounds rubric updates in longitudinal training evidence. AMARIS stores rollout analyses, step-level summaries, and rubric update records in a persistent evaluation memory, then retrieves recent and semantically relevant history to revise rubrics. We evaluate AMARIS across science, medicine, instruction following, and creative writing under both global and instance-specific rubric settings. AMARIS improves over static, local-adaptive, and memory-ablated baselines, such as +2.8 points on GPQA-Diamond and +2.2 points on IFBench over the strongest baselines, while analysis shows that memory reduces oscillatory rubric edits and supports a progression from early failure correction to later curriculum advancement. AMARIS runs asynchronously alongside the normal RL loop, reducing blocking latency relative to synchronous rubric updates.

强化学习评分系统记忆机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。