提出双通道机制，让多模态大模型持续学习时更抗遗忘。
Taming the Implicit: Dual-Channel Risk-Aware Reinforcement Fine-Tuning for Continual Multimodal Post-Training
- 双通道设计：分别控制更新强度和数据采样风险，显式管理优化风险。
- 在MLLM-CL上使最终遗忘率降低79.8%，同时保持新任务性能。
- 无需额外记忆，可无缝接入任意强化微调算法，适合持续学习场景。
强化微调（RFT）被认为能有效抵抗多模态大模型持续后训练中的灾难性遗忘。然而，在显著的任务分布偏移下，现有RFT算法的遗忘问题急剧加剧。这源于RFT固有的隐式奖励方差正则化，无法抑制失控的优化风险。本文提出首个显式风险治理的双通道框架——风险感知策略优化（RAPO）。在策略通道中，风险感知策略缩放通过回溯可靠性与基于Fisher信息的局部预测敏感性自适应调节样本级更新幅度；在数据通道中，风险感知动态分桶采样通过动态风险分层重组训练批次，引导优化聚焦于信息丰富且稳定的样本。作为即插即用策略，RAPO无需跨任务记忆，可无修改地适配任意RFT算法。在公开的MLLM-CL基准上，相比其RLOO基线，RAPO将最终遗忘率降低79.8%，同时保持对新任务的竞争力。
原文摘要 · Abstract (English)
Reinforcement fine-tuning (RFT) is widely believed to inherently resist catastrophic forgetting in continual post-training of multimodal large language models. Under pronounced task distributional shifts, however, forgetting across representative RFT algorithms escalates sharply. This stems from the implicit reward-variance regularization inherent to RFT, which proves incapable of suppressing uncontrolled optimization risk. We propose Risk-Aware Policy Optimization (RAPO), the first dual-channel framework for explicit risk governance in continual RFT. On the policy channel, Risk-Aware Policy Scaling adaptively calibrates per-sample update magnitude via rollout reliability and Fisher-inspired local predictive sensitivity; on the data channel, Risk-Aware Dynamic Bucket Sampling reorganizes training batches through dynamic risk stratification, steering optimization toward informative yet stable samples. As a plug-and-play strategy requiring no cross-task memory, RAPO generalizes to any RFT algorithm without modification. On the public MLLM-CL benchmark, RAPO reduces final forgetting by 79.8% relative to its RLOO backbone while retaining new-task competitiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。