arXiv:2607.17243cs.AI2026-07

用密集奖励控制推理长度,提升多视角空间推理准确率

LenGuard-GPC: Length Guarding with Guided-Prompt Consistency for Spatial Reasoning Reinforce Learning

论文配图:LenGuard-GPC: Length Guarding with Guided-Prompt Consistency for Spatial Reasoning Reinforce Learning
图 1 · 摘自论文原文
  • 通过对比标准提示与引导提示的逐标记预测分布,生成密集奖励信号
  • 在6个基准上实现更高准确率,平均响应长度减少30%以上
  • 适合需要精准且简洁推理的视觉语言任务研究者

多视角空间推理要求视觉-语言模型在图像间比较视觉证据、对齐物体对应关系,并在长视觉上下文中推断空间关系,而链式思维推理往往变得冗长却无精度提升。强化学习结合可验证奖励是该任务的自然选择,但标准GRPO奖励依赖稀疏的结果级反馈,无法提供推理轨迹出错位置的信号,也无法控制其长度。我们提出LenGuard-GPC,一种同时解决这两个问题的密集奖励框架。对于每条采样轨迹,它比较标准提示与引导提示下的逐标记预测分布,利用得到的逐标记KL散度之和作为密集奖励信号。由于该KL惩罚随标记累积,可能无差别奖励更短响应,因此引入分阶段长度奖励,在不单纯鼓励简短的前提下,将推理长度控制在合理范围。在六个多视角空间推理基准上,LenGuard-GPC在提升准确率的同时,降低平均响应长度。

原文摘要 · Abstract (English)

Multi-view spatial reasoning requires vision-language models to compare visual evidence across images, align object correspondences, and infer spatial relations over long visual contexts, a setting where chain-of-thought reasoning tends to grow verbose without becoming more accurate. Reinforcement learning with verifiable rewards is a natural fit for this task, but standard GRPO reward relies on sparse outcome-level feedback and gives no signal about where a reasoning trajectory goes wrong, nor any control over its length. We propose LenGuard-GPC, a dense reward framework that addresses both problems together. For each sampled trajectory, it compares the token-wise predictive distributions under a standard prompt and a guided prompt, and uses the resulting token-sum KL divergence as a dense reward signal. Since this KL penalty accumulates over tokens and would otherwise reward shorter responses regardless of their quality, we introduce a staged length bonus that keeps reasoning length within a controlled range without simply encouraging brevity. On six multi-view spatial reasoning benchmarks, LenGuard-GPC improves accuracy over vanilla GRPO while reducing average response length.

空间推理强化学习提示一致性长度控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。