用模型不确定度自动选关键帧，提升长视频理解效率与准确率
AdaptToken: Entropy-based Adaptive Token Selection for MLLM Long Video Understanding
- 基于模型响应熵动态分配每段视频的处理权重
- 在4个基准上平均提升6.7分，支持最长10000帧输入
- 无需训练即可实现早停，推理时间减半，适合长视频场景
长视频理解对多模态大语言模型（MLLM）仍具挑战，主要受限于高内存开销和上下文长度限制。现有方法通过评分筛选短片段内的帧或令牌，但缺乏跨远距离视频片段的可比性机制，也无法在证据充足时提前终止。我们提出AdaptToken，一种无需训练的框架，将MLLM的自不确定性转化为全局控制信号，用于长视频令牌选择。该方法将视频分组，提取跨模态注意力以排序每组内令牌，并利用模型响应熵估计每组提示的相关性。这一熵信号实现了组间令牌预算的全局分配，并支持早期停止（AdaptToken-Lite），在模型足够确定时跳过剩余组。在四个长视频基准（VideoMME、LongVideoBench、LVBench、MLVU）及多个基础MLLM（7B-72B）上，AdaptToken持续提升准确率（如在Qwen2.5-VL 7B上平均提升6.7分），并能有效处理长达10000帧的输入；AdaptToken-Lite则将推理时间减少约一半，性能相当。
原文摘要 · Abstract (English)
Long video understanding remains challenging for Multi-modal Large Language Models (MLLMs) due to high memory costs and context-length limits. Prior approaches mitigate this by scoring and selecting frames/tokens within short clips, but they lack a principled mechanism to (i) compare relevance across distant video clips and (ii) stop processing once sufficient evidence has been gathered. We propose AdaptToken, a training-free framework that turns an MLLM's self-uncertainty into a global control signal for long-video token selection. AdaptToken splits a video into groups, extracts cross-modal attention to rank tokens within each group, and uses the model's response entropy to estimate each group's prompt relevance. This entropy signal enables a global token budget allocation across groups and further supports early stopping (AdaptToken-Lite), skipping the remaining groups when the model becomes sufficiently certain. Across four long-video benchmarks (VideoMME, LongVideoBench, LVBench, and MLVU) and multiple base MLLMs (7B-72B), AdaptToken consistently improves accuracy (e.g., +6.7 on average over Qwen2.5-VL 7B) and continues to benefit from extremely long inputs (up to 10K frames), while AdaptToken-Lite reduces inference time by about half with comparable performance. Project page: https://haozheqi.github.io/adapt-token
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。