解决视频流中目标定位不一致问题,提升视觉定位稳定性与准确性。
TempoGround: State-Aware Streaming Visual Grounding with Vision-Language Models

- 通过状态感知的跨帧对应机制,追踪物体进出和持续状态。
- 在多个基准上提升2D/3D定位准确率,平均[email protected]提高4.4。
- 适合需要稳定实时视觉定位的自动驾驶与机器人场景。
视觉定位将语言指代映射到空间目标,是视觉语言模型实现开放词汇感知的核心任务。现有方法在单帧和视频视觉定位上取得进展,但在流式输入下仍面临身份漂移、跨帧不一致及部分遮挡下定位脆弱的问题。为此,我们提出TempoGround,一种原生基于视觉语言模型的框架,通过检测跨帧对象对应关系并显式建模对象存在状态,实现流式输入下的精准一致定位。其核心是状态感知的课程化预测机制:先解决2D实例关联,预测物体新进入、持续在场或离开视野,解码2D框后提升为相机坐标系下的3D框。由于仅依赖词元级监督无法捕捉流式定位的几何目标,我们进一步引入流式定位强化(SGR),通过可验证的定位、身份和一致性奖励联合优化,增强定位持续性与时间一致性。我们设计三阶段训练策略并在大规模数据上训练。在多个挑战性基准上评估流式输入下的视觉定位表现,TempoGround在[email protected]和[email protected]上平均提升4.4和0.5,在[email protected]和AP_3D上分别提升6.2和7.5。结果表明,TempoGround为流式输入下的视觉定位提供了实用基础。
原文摘要 · Abstract (English)
Visual grounding maps language referents to spatial targets and is central to open-vocabulary perception with vision-language models. Existing methods have made substantial progress on single-frame and video-based visual grounding, yet under streaming inputs they still suffer from identity drift, cross-frame inconsistency, and fragile localization under partial occlusion. To address these issues, we present TempoGround, a VLM-native framework that detects cross-frame object correspondence and explicitly models object presence states, thereby enabling accurate and consistent visual grounding under streaming inputs. The key is a curriculum prediction mechanism guided by state-aware cross-frame correspondence: TempoGround resolves 2D instance association, predicts whether each object newly enters, continues in, or leaves the view, decodes the 2D box, and then lifts it to a camera-frame 3D box. As token-level supervision alone cannot capture the geometric objectives of streaming grounding, we further introduce Streaming Grounding Reinforcement (SGR), which optimizes TempoGround with verifiable Grounding, Identity, and Consistency rewards, jointly reinforcing persistent localization and temporally consistent predictions. We carefully design a three-stage training strategy and train TempoGround on large-scale data. We evaluate visual grounding under causally streaming inputs on multiple challenging benchmarks: TempoGround improves [email protected] and [email protected] by 4.4 and 0.5 on average, and [email protected] and AP_3D by 6.2 and 7.5, respectively. These results demonstrate that TempoGround provides a practical foundation for visual grounding under streaming inputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。