让AI用连续坐标推理图像,定位更准、训练更快。
Thinking with Images as Continuous Actions: Numerical Visual Chain-of-Thought
- 用连续坐标代替离散标签,直接生成框位置
- 在三个数据集上定位精度提升,答案准确率超基线
- 支持强化学习,适合做视觉推理的模型开发者
当前多模态大模型依赖视觉思维链进行图像区域推理,但现有方法或使用文本化坐标导致模态错配与语义断裂,或采用固定粒度块,限制精确定位且需复杂架构修改。本文提出数值化视觉思维链(NV-CoT),将多模态大模型的动作空间从离散词汇扩展到连续欧氏空间,使模型可仅通过微小架构调整直接生成边界框坐标作为动作。该框架支持监督微调与强化学习,将分类策略替换为坐标上的高斯(或拉普拉斯)分布策略,并通过重参数化采样引入随机性,兼容GRPO类策略优化。在三个基准测试中对比八种代表性视觉推理基线,实验表明NV-CoT显著提升定位精度与最终答案准确率,同时加速训练收敛,验证了连续动作视觉推理的有效性。代码已开源。
原文摘要 · Abstract (English)
Recent multimodal large language models (MLLMs) increasingly rely on visual chain-of-thought to perform region-grounded reasoning over images. However, existing approaches ground regions via either textified coordinates-causing modality mismatch and semantic fragmentation or fixed-granularity patches that both limit precise region selection and often require non-trivial architectural changes. In this paper, we propose Numerical Visual Chain-of-Thought (NV-CoT), a framework that enables MLLMs to reason over images using continuous numerical coordinates. NV-CoT expands the MLLM action space from discrete vocabulary tokens to a continuous Euclidean space, allowing models to directly generate bounding-box coordinates as actions with only minimal architectural modification. The framework supports both supervised fine-tuning and reinforcement learning. In particular, we replace categorical token policies with a Gaussian (or Laplace) policy over coordinates and introduce stochasticity via reparameterized sampling, making NV-CoT fully compatible with GRPO-style policy optimization. Extensive experiments on three benchmarks against eight representative visual reasoning baselines demonstrate that NV-CoT significantly improves localization precision and final answer accuracy, while also accelerating training convergence, validating the effectiveness of continuous-action visual reasoning in MLLMs. The code is available in https://github.com/kesenzhao/NV-CoT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。