通过在隐空间注入噪声,提升视觉语言模型的推理能力。
Perturb the Thought, Not the Pixels: Latent-Space Rollout Diversification for Reinforcement Learning of Vision-Language Models

- 在提示编码的最后一层注入高斯噪声,实现隐空间多样化
- 在5个外部基准上显著提升数学推理准确率(p≤0.001)
- 适用于各类视觉语言模型,仅需少量代码修改
基于可验证奖励的强化学习(RLVR)能提升视觉语言模型(VLMs)的推理能力,而优化组内轨迹多样化可进一步放大效果。现有方法通过解码温度或像素空间图像扰动实现多样化;我们提出将扰动引入模型隐空间。提出噪声对比式GRPO(NC-GRPO),在每组轨迹中一半的前向传播中,向提示编码的最后隐藏层注入尺度校准的高斯噪声,使这些轨迹从偏移的初始状态分支。成功抵达答案的分支会被强化,而偏离的则被抑制,从而将分支点敏感性转化为策略梯度信号;目标函数、奖励和推理流程保持不变。在几何3K数据集上训练的Qwen2.5-VL-7B模型上,与原始GRPO相比,NC-GRPO在五个外部基准上显著提升跨域数学推理能力(合并麦克内马尔检验p≤0.001),同时提高域内准确率和幻觉鲁棒性——后者是像素空间噪声反而退化的维度。机制消融表明,独立随机多样性才是关键,而非噪声预算或方向;噪声尺度研究揭示了推理专精与通用能力之间的权衡。NC-GRPO设计为模态无关,仅需约50行代码修改即可集成至标准RLVR流水线。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) improves the reasoning ability of vision-language models (VLMs), and diversifying the rollouts within each optimization group amplifies its gains. Existing approaches diversify through decoding temperature or pixel-space image distortion; we ask whether the perturbation belongs in the model's latent space instead. We introduce Noise-Contrastive GRPO (NC-GRPO), which injects scale-calibrated Gaussian noise into the last hidden layer of the prompt-encoding pass for half of each rollout group, branching those rollouts from a displaced departure state. Branches that reach the answer despite the displacement are reinforced over those derailed by it, converting sensitivity at the branch point into policy-gradient signal; the objective, reward, and inference protocol are untouched. On Qwen2.5-VL-7B trained on Geometry3K, NC-GRPO significantly improves out-of-domain mathematical reasoning over vanilla GRPO across five held-out benchmarks (pooled McNemar $p \le 0.001$) while also improving in-domain accuracy and hallucination robustness -- the latter an axis on which image-space noise regresses even while posting a larger OOD average on perception-heavy benchmarks. Mechanism ablations indicate that independent stochastic diversity, not noise budget or direction, is the active ingredient, and a noise-scale study exposes a dial between reasoning specialization and general capability. NC-GRPO is designed to be modality-agnostic and integrates into a standard RLVR pipeline as a ~50-line change to the inference engine.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。