通过并行处理视频帧子集提升模型感知能力
Video Parallel Scaling: Aggregating Diverse Frame Subsets for VideoLLMs
- 多路并行推理，每路处理不同帧子集
- 在多个基准上性能显著提升，2B-32B模型均有效
- 无需训练，适合增强视频大模型的时序推理
视频大语言模型（VideoLLMs）面临核心瓶颈：增加输入帧数以捕捉精细时序细节会带来高昂计算成本和长上下文导致的性能下降。我们提出视频并行扩展（VPS），一种推理阶段方法，在不增加上下文窗口的前提下扩大模型感知带宽。VPS通过运行多个并行推理流，每路处理视频中互不重叠的帧子集，再聚合各流输出概率，整合更丰富的视觉信息。理论上，该方法通过利用无关视觉证据，有效压缩了Chinchilla缩放定律，提升性能而无需额外训练。在Video-MME、EventHallusion等基准上，针对2B至32B规模的多种模型架构进行大量实验表明，VPS持续且显著提升性能，优于其他并行策略（如Self-consistency），且与其它解码策略互补，提供了一种高效、鲁棒的框架，增强VideoLLMs的时序推理能力。
原文摘要 · Abstract (English)
Video Large Language Models (VideoLLMs) face a critical bottleneck: increasing the number of input frames to capture fine-grained temporal detail leads to prohibitive computational costs and performance degradation from long context lengths. We introduce Video Parallel Scaling (VPS), an inference-time method that expands a model's perceptual bandwidth without increasing its context window. VPS operates by running multiple parallel inference streams, each processing a unique, disjoint subset of the video's frames. By aggregating the output probabilities from these complementary streams, VPS integrates a richer set of visual information than is possible with a single pass. We theoretically show that this approach effectively contracts the Chinchilla scaling law by leveraging uncorrelated visual evidence, thereby improving performance without additional training. Extensive experiments across various model architectures and scales (2B-32B) on benchmarks such as Video-MME and EventHallusion demonstrate that VPS consistently and significantly improves performance. It scales more favorably than other parallel alternatives (e.g. Self-consistency) and is complementary to other decoding strategies, offering a memory-efficient and robust framework for enhancing the temporal reasoning capabilities of VideoLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。