arXiv:2506.04034cs.CV2025-06被引 24

让视觉指代模型像人一样一步步推理,提升准确性和可信度。

Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning

  • 将指代任务转为分步推理,逐个检验候选物体匹配度。
  • 在标准数据集上精度更高,且能有效拒绝无匹配的描述。
  • 适合需要可解释性与鲁棒性的实际应用,如自动驾驶、医疗影像。

视觉指代旨在识别图像中符合自然语言描述的所有物体。我们提出,一个可靠的指代模型应具备可落地性:预测结果需可验证(有可解释的推理过程并关联视觉证据)且可信(当无匹配物体时主动拒绝)。现有方法多直接预测边界框,缺乏解释性,难以处理无匹配情况。本文提出Rex-Thinker,将指代建模为显式的链式思维(CoT)推理任务:先定位目标类别候选对象,再对每个候选进行逐步推理判断是否匹配表达,最后输出结果。为此,我们基于HumanRef数据集,通过GPT-4o构建大规模CoT风格数据集HumanRef-CoT,每条推理遵循规划-动作-总结结构,支持模型学习分解式可解释推理。训练采用两阶段策略:先监督微调建立推理能力,再用GRPO强化学习优化精度与泛化性。实验表明,该方法在域内评估中同时提升精度与可解释性,并在域外场景中展现更强拒答能力与泛化性能。

原文摘要 · Abstract (English)

Object referring aims to detect all objects in an image that match a given natural language description. We argue that a robust object referring model should be grounded, meaning its predictions should be both explainable and faithful to the visual content. Specifically, it should satisfy two key properties: 1) Verifiable, by producing interpretable reasoning that justifies its predictions and clearly links them to visual evidence; and 2) Trustworthy, by learning to abstain when no object in the image satisfies the given expression. However, most methods treat referring as a direct bounding box prediction task, offering limited interpretability and struggling to reject expressions with no matching object. In this work, we propose Rex-Thinker, a model that formulates object referring as an explicit CoT reasoning task. Given a referring expression, we first identify all candidate object instances corresponding to the referred object category. Rex-Thinker then performs step-by-step reasoning over each candidate to assess whether it matches the given expression, before making a final prediction. To support this paradigm, we construct a large-scale CoT-style referring dataset named HumanRef-CoT by prompting GPT-4o on the HumanRef dataset. Each reasoning trace follows a structured planning, action, and summarization format, enabling the model to learn decomposed, interpretable reasoning over object candidates. We then train Rex-Thinker in two stages: a cold-start supervised fine-tuning phase to teach the model how to perform structured reasoning, followed by GRPO-based RL learning to improve accuracy and generalization. Experiments show that our approach outperforms standard baselines in both precision and interpretability on in-domain evaluation, while also demonstrating improved ability to reject hallucinated outputs and strong generalization in out-of-domain settings.

视觉指代链式思维可解释性推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。