研究长视频中关键帧选择如何影响多模态大模型表现,发现选帧比压缩更重要。
Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs
- 分三步实验:选帧、压缩、重投入,控制变量验证各环节作用
- 选8帧比选16帧高6.9分,稀疏算法性能媲美专门设计的选帧器
- 压缩节省的资源若用于更多帧,能再提升2-3分,效果才真正显现
长视频语言模型无法处理每一帧:一小时视频每秒采样一次即达3600张图像,系统仅保留少量固定帧。通常将帧选择视为预处理细节,本文质疑其合理性。现有选帧方法常同时改变评分器、提示边界、分辨率策略和回答模型,难以公平比较。我们保持这些因素不变,仅逐项测试选择、空间压缩与节省资源的重投入,在六种无训练选帧规则、三个长视频基准和两个回答模型上进行实验。结果表明,选择是最大影响因素:在LongVideoBench的一小时视频任务中,用查询选定的8帧比均匀选取的16帧高出6.9分;未经修改的稀疏逼近算法OMP,在所有三个基准上均达到或接近专为该任务设计的选帧器性能。压缩几乎无代价:在固定时间点将每帧空间预算减半,最多损失0.44分。但只有将节省的令牌重新投入生成更多压缩帧时,压缩才真正带来增益——以双倍数量压缩帧替代原8帧,成本不高于原方案,准确率进一步提升2至3分。此外,实验揭示了自身基线中的实现缺陷及不同评估框架间存在0.07至3.74分差距,凸显统一受控环境的重要性。
原文摘要 · Abstract (English)
Long-video language models cannot look at every frame: an hour sampled once per second is 3,600 images, and a system keeps only a small fixed slice of that pool. Which frames survive that slice is usually treated as a preprocessing detail; we test whether it should be. Published selectors make the comparison hard because they change the frame scorer, the prompt boundary, the resolution policy, and the answering model all at once. We hold each fixed and vary one decision at a time: selection, spatial compression, and reinvestment of the savings, across six training-free selection rules, three long-video benchmarks, and two answering models. Selection is the largest single lever: on LongVideoBench's hour-long bin, eight query-selected frames beat sixteen uniformly spaced ones by 6.9 points, and Orthogonal Matching Pursuit, an unmodified decades-old sparse-approximation algorithm, matches or comes within a point of every purpose-built selector we compare it against, across all three benchmarks. Compression is close to free: halving each frame's spatial budget at fixed timestamps costs at most 0.44 points. Reinvestment is where that budget turns back into accuracy: spending the freed tokens on twice as many compressed frames, at a measured cost no higher than the original eight, returns a further two to three points; compression only pays off once its savings are spent this way. Along the way, an implementation bug in our own AKS baseline and a 0.07 to 3.74 point gap between two harnesses running the same published rules at the same budget show why these comparisons need to happen inside one controlled harness rather than across papers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。