arXiv:2510.21978cs.LGcs.AI2025-10被引 3

提出动态重加权策略,防止大模型在强化学习后遗忘基础能力。

Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models

  • 通过在线监测收敛与波动,动态调整不同任务的训练权重。
  • 在Qwen2.5-VL系列模型上,既保持感知与忠实性,又提升推理性能。
  • 无需额外训练,可直接接入现有强化学习流程,适合模型优化者使用。

基于可验证奖励的强化学习(RLVR)在数学和多模态推理方面取得了显著进展,已成为当代语言与视觉语言模型的标准后训练范式。然而,该方法存在严重的能力退化风险:模型在长期训练中会遗忘基础技能,若不采用正则化策略。我们实证发现,开源推理模型在感知与忠实性等核心能力上出现性能下降。虽然使用KL散度等正则项可防止偏离基础模型,但这些项仅针对当前任务计算,无法保障更广泛知识的保留。同时,跨异构领域的经验回放使得各目标的训练权重难以确定。为此,我们提出RECAP——一种具备动态目标重加权机制的经验回放策略,以实现通用知识的持续保留。其重加权机制通过短期收敛与不稳定性信号在线调整,将训练重点从饱和任务转向表现不佳或波动较大的任务。该方法为端到端设计,可无缝集成至现有RLVR流程,无需额外训练或复杂调参。在Qwen2.5-VL-3B和Qwen2.5-VL-7B上的大量实验表明,该方法不仅能有效保留通用能力,还能通过更灵活的任务间权衡提升推理表现。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards (RLVR) has delivered impressive gains in mathematical and multimodal reasoning and has become a standard post-training paradigm for contemporary language and vision-language models. However, the RLVR recipe introduces a significant risk of capability regression, in which models forget foundational skills after prolonged training without employing regularization strategies. We empirically confirm this concern, observing that open-source reasoning models suffer performance degradation on core capabilities such as perception and faithfulness. While imposing regularization terms like KL divergence can help prevent deviation from the base model, these terms are computed on the current task and therefore do not guarantee preservation of broader knowledge. Meanwhile, commonly used experience replay across heterogeneous domains makes it nontrivial to decide how much training emphasis each objective should receive. To address this, we propose RECAP-a replay strategy with dynamic objective reweighting for general knowledge preservation. Our reweighting mechanism adapts online using short-horizon signals of convergence and instability, shifting the post-training focus away from saturated objectives and toward underperforming or volatile ones. Our method is end-to-end and readily applicable to existing RLVR pipelines without training additional models or heavy tuning. Extensive experiments on benchmarks using Qwen2.5-VL-3B and Qwen2.5-VL-7B demonstrate the effectiveness of our method, which not only preserves general capabilities but also improves reasoning by enabling more flexible trade-offs among in-task rewards.

大模型强化学习知识保持推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。