动态调整视频采样精度，兼顾细节与长程上下文。
LENS: Adaptive Spatio-Temporal Zooming for Keyframe Sampling in Long-Form Videos
- 根据文本查询自动决定局部放大或全局覆盖
- 在相同帧数下将视频理解准确率提升至60.7%
- 无需训练，适合长视频任务的高效采样
尽管多模态大模型进展迅速，长视频理解仍受限于有限的上下文窗口。现有关键帧采样方法虽能将视频压缩为精炼帧集，但时空搜索空间庞大，空间细节与时间覆盖常存在冲突。为此，我们提出LENS，一种无需训练的关键帧采样框架，可依据文本查询动态决定何时聚焦局部细节（空间放大），何时扩展时间范围（时间缩放），从而在有限帧预算下实现多粒度推理。该方法通过帧内区域强调与多帧聚合结合，同时捕捉高保真细节与长程上下文。在多个长视频基准测试中，LENS持续优于现有最先进方法，并显著超越均匀采样，在Video-MME上将准确率从53.3%提升至60.7%（使用Qwen2.5-VL）。代码已开源：https://github.com/zhangce01/LENS。
原文摘要 · Abstract (English)
Despite rapid progress in Multi-modal Large Language Models (MLLMs), understanding long-form videos is still bottlenecked by limited context windows. While recent keyframe sampling methods attempt to mitigate this by distilling video inputs into a compact set of query-relevant frames, navigating the vast spatio-temporal search space remains challenging, as spatial detail and temporal coverage often conflict. To address this, we introduce LENS, a training-free keyframe sampling framework that dynamically decides when to zoom in for fine-grained details and when to zoom out for broader context based on the text query. Concretely, LENS adaptively allocates a limited frame budget between spatial zoom-ins, which highlight query-relevant regions within individual frames, and temporal zoom-outs, which expand the temporal scope through multi-frame aggregation, enabling the model to reason across multiple granularities while capturing both high-fidelity details and long-range context. Across diverse long-form video benchmarks, LENS consistently outperforms prior state-of-the-art keyframe sampling methods and delivers substantial gains over uniform sampling, improving Video-MME accuracy from 53.3% to 60.7% with Qwen2.5-VL.Code is available at https://github.com/zhangce01/LENS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。