arXiv:2607.24570cs.CVcs.AI2026-07

针对视频稀疏帧输入,仅微调模型末尾部分视觉层可大幅提升定位准确率。

The Visual Bottleneck: Sparse-Frame Adaptation of MLLMs for Joint Spatial-Temporal Video Grounding

论文配图:The Visual Bottleneck: Sparse-Frame Adaptation of MLLMs for Joint Spatial-Temporal Video Grounding
图 1 · 摘自论文原文
  • 只微调视觉模型最后3层(占4%参数),显著提升稀疏帧下的定位性能
  • 在16帧输入下,模型达到68.8%的时序mIoU,优于零样本8B模型12.8点
  • 边界感知采样策略可再提升26点,适合实际部署中的视频审核场景

大规模视频平台每小时处理数百万条上传内容,需定位违规行为发生的时间与位置。因计算成本限制,系统通常仅使用每视频8至16帧的稀疏输入,但现有多模态大模型(MLLM)在数百帧密集序列上预训练,导致训练与部署条件严重不匹配。例如,Qwen3-VL 8B模型在帧数从稠密降至16帧时,时序mIoU从56.0%暴跌至22.3%,相对下降60.2%。本文系统研究了多种训练策略以弥合该差距。结果表明,视觉特征提取是稀疏输入下的主要瓶颈。仅微调最后三层ViT(占总参数4%),即可实现68.8%的时序mIoU,超越使用稠密输入的零样本8B模型12.8点。语言模型微调则几乎无益甚至有害。结合边界信息的Hybrid16采样策略,在有时间边界可用时,相较均匀采样进一步提升26点。结论:对于稀疏帧视频定位任务,训练策略比模型规模更重要——经微调的2B模型始终优于未微调的8B模型,无论是否具备稠密帧访问能力。

原文摘要 · Abstract (English)

Large-scale video platforms process millions of uploads hourly, requiring moderation systems that can localize when and where policy violations occur within each video. Processing every frame is infeasible at scale, so systems are constrained to sparse inputs of 8 to 16 frames per video. Yet state-of-the-art multimodal large language models (MLLMs) are pretrained on dense sequences of hundreds of frames, creating a fundamental mismatch between training and deployment conditions. This mismatch causes severe performance collapse: the Qwen3-VL 8B model drops from 56.0% to 22.3% temporal mIoU when frames are reduced to 16, a 60.2% relative degradation. We present a systematic empirical study of training strategies to close this gap for spatial-temporal video grounding. Our results suggest that visual feature extraction is the dominant bottleneck under sparse-frame inputs. Adapting only the final three ViT layers, 4% of total parameters, achieves 68.8% temporal mIoU and surpasses a zero-shot 8B model using dense inputs by 12.8 points. Language model fine-tuning, by contrast, offers negligible or negative returns. A boundary-aware sampling strategy, Hybrid16, further improves temporal mIoU by 26 points over uniform sampling when temporal boundaries are available. We conclude that for sparse-frame video grounding, training strategy dominates model scale: a fine-tuned 2B model consistently outperforms a zero-shot 8B model, with or without dense frame access.

视频定位稀疏帧视觉模型高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。