arXiv:2601.07695cs.CV2026-01被引 1

提出平滑奖励机制,让视觉语言模型更准地理解3D空间关系。

Smooth Operator: Smooth Verifiable Reward Activates Spatial Reasoning Ability of Vision-Language Model

  • 用动态Sigmoid函数生成连续奖励信号,解决传统方法奖励稀疏问题。
  • 在5万条3D任务上验证,性能媲美大规模监督方法,数据效率更高。
  • 适合研究3D理解、具身智能或强化学习的开发者使用。

视觉语言模型在实现3D场景理解的精确数值预测时面临瓶颈。传统强化学习方法依赖相对排序,常因奖励稀疏和梯度不稳,无法有效利用3D物理约束提供的可验证信号。标准GRPO框架中,相对归一化导致误差微小但非零的‘近似样本’出现优势崩溃,造成宝贵边界样本在优化中被丢弃。为此,本文提出平滑数值奖励激活(SNRA)算子与绝对保持型GRPO(AP-GRPO)框架。SNRA采用动态参数化Sigmoid函数,将原始反馈转化为密集连续的奖励流;AP-GRPO引入绝对标量梯度,缓解传统相对排序中的数值信息损失。基于此构建了包含50,000个可验证3D子任务的Numerical3D-50k数据集。实验表明,AP-GRPO在性能上达到大规模监督方法水平,同时具备更高数据效率,无需修改模型结构即可激活视觉语言模型的潜在3D推理能力。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) face a critical bottleneck in achieving precise numerical prediction for 3D scene understanding. Traditional reinforcement learning (RL) approaches, primarily based on relative ranking, often suffer from severe reward sparsity and gradient instability, failing to effectively exploit the verifiable signals provided by 3D physical constraints. Notably, in standard GRPO frameworks, relative normalization causes "near-miss" samples (characterized by small but non-zero errors) to suffer from advantage collapse. This leads to a severe data utilization bottleneck where valuable boundary samples are discarded during optimization. To address this, we introduce the Smooth Numerical Reward Activation (SNRA) operator and the Absolute-Preserving GRPO (AP-GRPO) framework. SNRA employs a dynamically parameterized Sigmoid function to transform raw feedback into a dense, continuous reward continuum. Concurrently, AP-GRPO integrates absolute scalar gradients to mitigate the numerical information loss inherent in conventional relative-ranking mechanisms. By leveraging this approach, we constructed Numerical3D-50k, a dataset comprising 50,000 verifiable 3D subtasks. Empirical results indicate that AP-GRPO achieves performance parity with large-scale supervised methods while maintaining higher data efficiency, effectively activating latent 3D reasoning in VLMs without requiring architectural modifications.

3D理解强化学习视觉语言模型奖励设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。