用视觉提示直接注入目标框信息,提升视频理解模型的精度与效率。
BoxTuning: Directly Injecting the Object Box for Multimodal Model Fine-Tuning
- 将目标框和轨迹以颜色图像形式注入视频,仅保留简短文字说明。
- 文本令牌减少87%-93%,保持全时序分辨率,捕捉运动方向与速度。
- 适用于需要精细空间推理的视频问答任务,尤其适合多模态大模型优化。
物体级时空理解对视频问答至关重要,但现有多模态大语言模型(MLLMs)整体编码帧,缺乏细粒度目标定位机制。近期工作将边界框坐标序列化为文本标记,但该文本-坐标范式存在根本性模态错配:目标信息本为视觉属性,却以文本形式编码,导致高令牌开销,迫使激进的时间降采样。我们提出BoxTuning,通过将物体时空信息直接注入视觉模态解决此问题。将彩色边界框与轨迹线渲染至视频帧作为视觉提示,仅保留简洁的颜色-对象图例作为文本。实际中实现87%-93%的文本令牌减少,同时保留完整时间分辨率;轨迹线在关键帧内进一步编码帧间运动方向与速度,恢复了文本坐标方法被迫丢弃的细粒度动态信息。在五个视频问答基准(CLEVRER、Perception Test、STAR、NExT-QA、IntentQA)上的实验表明,BoxTuning在空间导向任务上超越文本坐标基线,在推理主导任务上几乎消除准确率下降,确立视觉提示是向视频MLLM传递物体信息更自然、高效的新范式。
原文摘要 · Abstract (English)
Object-level spatial-temporal understanding is essential for video question answering, yet existing multimodal large language models (MLLMs) encode frames holistically and lack explicit mechanisms for fine-grained object grounding. Recent work addresses this by serializing bounding box coordinates as text tokens, but this text-coordinate paradigm suffers from a fundamental modality mismatch: object information is inherently visual, yet encoding it as text incurs a high token cost that forces aggressive temporal downsampling. We propose BoxTuning, which resolves this mismatch by injecting object spatial-temporal information directly into the visual modality. Colored bounding boxes and trajectory trails are rendered onto video frames as visual prompts, with only a concise color-to-object legend retained as text. This reduces the token cost significantly, achieving 87-93% text token reduction in practice. It also preserves full temporal resolution, where the trajectory trails further encode inter-frame motion direction and speed within each keyframe, recovering fine-grained dynamics that text-coordinate methods are forced to discard. Experimental results on five video QA benchmarks (CLEVRER, Perception Test, STAR, NExT-QA, IntentQA) show that BoxTuning surpasses text-coordinate baselines on spatially oriented tasks and nearly eliminates the accuracy degradation observed on reasoning-centric tasks, establishing visual prompting as a more natural and efficient paradigm for conveying object information to video MLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。