让模型用自己说话方式推理,提升视频理解能力。
Reason in the Words You Speak: Idiolectal Paraphrasing Off-Policy Traces for Reasoning Distillation in VideoLLMs

- 将教师模型的推理路径改写为学生模型的个性化表达
- 在5个基准上对3个大模型均实现稳定性能提升
- 可通用插件形式增强强化学习与监督微调
近期大型语言模型在复杂推理任务中表现优异,基于组相对策略优化(GRPO)的强化学习成为主流训练范式。然而,GRPO的在线策略特性限制了模型只能学习已有推理能力,难以掌握更高级技能。先前方法通过引入更强教师模型的特权推理轨迹指导训练,但这些轨迹与学生策略分布不一致。我们发现这种分布偏差导致语义关键推理标记的梯度截断,虽奖励正确答案,却未有效学习其解释过程。为此,提出Echo-GRPO框架,使模型以自身特有的词汇和表达习惯(即'个人语体')重构推理路径,同时通过双参考解码保持原意。该框架实例化为VideoEcho-R1,在三个多模态大模型骨干和五个基准上实现一致改进。进一步证明,该个人语体重写模块可作为通用插件,显著提升强化学习与监督微调框架的推理蒸馏效果,表明策略对齐监督具有普适性。
原文摘要 · Abstract (English)
Recent large language models achieve strong performance on complex reasoning tasks, where reinforcement learning with Group Relative Policy Optimization (GRPO) has emerged as a leading paradigm for optimizing models on self-generated trajectories. However, the on-policy nature of GRPO bounds the model to the reasoning skills it can already produce, restricting to learn more advanced capabilities. Prior works inject privileged reasoning traces from a stronger teacher policy to guide training, yet these traces are inherently out of distribution with respect to the student policy. We observe that this mismatch between on-policy and off-policy causes gradient clipping on semantically critical reasoning tokens, ultimately rewarding correct answers while leaving the reasoning that justifies them unlearned. Hence, we propose \textbf{Echo-GRPO}, a framework that lets the model reason in the words it speaks. Rather than imitating low-probability privileged traces from the teacher model, Echo-GRPO rewrites them into the student policy's own \textit{idiolect}, that is, its own characteristic vocabulary and expression patterns, while preserving their semantics via Dual-Reference Decoding. We instantiate this framework as \textbf{VideoEcho-R1} for video reasoning distillation, achieving consistent improvements across three multimodal LLM backbones and five benchmarks. Finally, we show that our idiolectal paraphrasing is a plug-in module that consistently improves both RL and supervised fine-tuning frameworks for reasoning distillation, demonstrating that policy-aligned supervision extends beyond GRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。