用分步推理提升模型对物体局部的定位能力
Reasoning-Guided Part-Level Visual Grounding via Reinforcement Learning
- 先定位整体物体，再在其中精确定位局部部件
- 40亿参数模型超越70亿参数模型和SAM3
- 适合需要精细视觉定位的多模态任务
多模态大语言模型能良好地从自然语言查询中定位完整物体，但在查询涉及物体部件时表现不佳。我们发现这是因为缺乏物体-部件的层级结构，导致部件与整体在同一阶段被定位。为此提出物体-部件层级反射定位（OP-HRG）策略：先粗略定位父物体，再在其内部精确定位部件；通过自检机制反思结果，并扩展重编码预测裁剪区域以验证修正效果。引入部件感知的GRPO框架，分阶段奖励训练该流程。一个40亿参数模型在PascalPart、PartImageNet和InstructPart数据集上优于70亿参数的基线模型和SAM3，且具备推理分割的迁移能力。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) ground whole objects well from free-form language queries, but they struggle when the query names a part rather than the object. We trace this to a missing object-part hierarchy, since parts are localized in the same single step used for objects. We propose Object-Part Hierarchical Reflective Grounding (OP-HRG), a coarse-to-fine reasoning-guided grounding strategy that first localizes the parent object and then the part within it. A self-check then reflects on the result, with an extension to re-encode the predicted crop to inspect the region it is correcting. We introduce a part-aware GRPO framework to train our pipeline with stage-wise rewards. A 4B model trained this way outperforms 7B grounding LLMs and SAM3 across PascalPart, PartImageNet, and InstructPart, and transfers to reasoning segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。