Triage通过分层视觉预算，高效处理视频推理中的冗余问题。
Triage: Hierarchical Visual Budgeting for Efficient Video Reasoning in Vision-Language Models
- 分两阶段：先选关键帧，再分配核心与上下文标记
- 在多个基准上实现更快推理、更低内存占用且性能不降
- 无需训练，可直接接入现有视觉语言模型
视觉语言模型（VLMs）在视频处理中面临严重计算挑战，源于数据冗余导致的过长标记序列。为此，我们提出Triage——一种无需训练、即插即用的框架，将视频推理重构为资源分配问题，通过分层视觉预算实现高效处理。第一阶段‘帧级预算’基于视觉动态和相关性评估识别关键帧，生成重要性评分作为战略先验；第二阶段‘标记级预算’分两步分配标记：首先保留高相关性核心标记，随后使用高效的批量最大边际相关性（MMR）算法选取多样化上下文标记。大量实验表明，Triage在多个视频推理基准上显著提升推理速度、降低内存开销，同时保持或超越基线方法性能。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) face significant computational challenges in video processing due to massive data redundancy, which creates prohibitively long token sequences. To address this, we introduce Triage, a training-free, plug-and-play framework that reframes video reasoning as a resource allocation problem via hierarchical visual budgeting. Its first stage, Frame-Level Budgeting, identifies keyframes by evaluating their visual dynamics and relevance, generating a strategic prior based on their importance scores. Guided by this prior, the second stage, Token-Level Budgeting, allocates tokens in two phases: it first secures high-relevance Core Tokens, followed by diverse Context Tokens selected with an efficient batched Maximal Marginal Relevance (MMR) algorithm. Extensive experiments demonstrate that Triage improves inference speed and reduces memory footprint, while maintaining or surpassing the performance of baselines and other methods on various video reasoning benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。