对比三种视频时间定位输出方式,发现连续时序解码最高效准确。
How Should Video LLMs Output Time? An Analysis of Efficient Temporal Grounding Paradigms
- 统一模型和训练条件,比较文本数字、时序标记、连续解码三种输出方式。
- 连续解码在定位精度与延迟间平衡最优,推理延迟最低。
- 适合资源受限场景部署,为边缘计算视频理解提供实证指导。
虽然多模态大语言模型(MLLM)推动了视频时间定位(VTG)发展,但现有方法常将输出范式与不同骨干网络、数据集及训练协议耦合,难以分离输出设计的影响。随着VTG系统向资源受限的边缘设备部署演进,输出形式与系统效率的权衡亟需系统研究。本文通过受控实验,对比三种主流VTG输出范式:文本数字生成、时序标记生成与连续时序解码。在相同紧凑视觉语言模型(SmolVLM2、FastVLM、Molmo2)上,使用一致数据集和LoRA微调协议进行评估。在Charades-STA、QVHighlights、YouCook2上的测试涵盖定位精度与系统效率(推理延迟、训练吞吐量、参数开销)。结果表明,输出形式的选择显著影响定位精度与计算成本,且独立于模型规模。连续分布范式始终在帕累托前沿表现最优,实现高鲁棒性定位的同时最小化延迟开销。研究为设计高效可部署的VTG系统提供了客观实证依据。
原文摘要 · Abstract (English)
While Multimodal Large Language Models (MLLMs) have advanced Video Temporal Grounding (VTG), existing methods often couple output paradigms with different backbones, datasets, and training protocols. This makes it challenging to isolate the specific impact of the output design. Additionally, as VTG systems are increasingly considered for resource-constrained edge deployment, the trade-off between output formulation and system-level efficiency requires systematic investigation. In this paper, we present a controlled empirical study comparing three dominant VTG output paradigms: Text Numeral Generation, Temporal Token Generation, and Continuous Temporal Decoding. We evaluate these paradigms across identical compact VLMs (SmolVLM2, FastVLM, and Molmo2) using consistent datasets and LoRA fine-tuning protocols. Evaluations on Charades-STA, QVHighlights, and YouCook2 measure both localization accuracy and system efficiency, including inference latency, training throughput, and parameter overhead. Our results demonstrate that the choice of output formulation significantly affects both grounding accuracy and computational cost, independent of model scale. Specifically, the continuous distribution paradigm consistently achieves the most favorable efficiency-accuracy trade-off on the Pareto frontier, delivering robust localization with minimal latency overhead. These findings provide objective empirical guidelines for designing efficient, deployment-ready VTG systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。