arXiv:2608.22359cs.CVcs.AI2026-08

用音频提前筛选视频片段,大幅减少昂贵的视觉语言模型调用次数。

Pre-Decoding Acoustic Triage for Budgeted Vision-Language Captioning of Untrimmed Egocentric Video

论文配图:Pre-Decoding Acoustic Triage for Budgeted Vision-Language Captioning of Untrimmed Egocentric Video
图 1 · 摘自论文原文
  • 先用音频特征筛选需处理的视频片段,无需解码视频帧即可决策。
  • 在相同动作覆盖率下,减少超过一半的VLM调用,最高降20%。
  • 适合资源受限场景,如物流、制造中的长时第一人称视频分析。

自动分析长达数小时的第一人称视频对物流、建筑和制造领域的进度监控、质量控制与安全至关重要。现有方法使用视觉语言模型(VLM)处理固定窗口,但成本随模型调用次数线性增长。以往的筛选策略或随机采样,或依赖视觉特征排序,却需预先解码视频,违背预算限制。本文提出音频优先筛选:利用最轻量的音频模态,在解码任何视频帧前即完成窗口选择,可自然结合令牌压缩或量化。核心创新在于目标设定——不检测每帧声音事件,而是训练选择器在每个动作发生时触发一次。该设计在所有评估调用率下提升动作覆盖4.0-10.8个百分点,仅使用冻结的AudioSet预训练特征,无需领域特定的声音标签。在EPIC-KITCHENS-100上,调用次数不足一半时仍保持覆盖率,减少9-20%的VLM调用;在Ego4D的247个片段上优于均匀采样与两种近期视觉关键帧选择器。代码与所有结果文件见https://github.com/masjalayer/PreDecoding-AcousticTriage。

原文摘要 · Abstract (English)

Automatically analyzing hours-long egocentric video is increasingly essential for progress monitoring, quality control, and safety in logistics, construction, and manufacturing. Yet current pipelines that process short, fixed-size windows with a vision-language model (VLM) are prohibitively expensive because cost scales with the number of model calls. To reduce this cost, prior work proposes triage policies to select which windows merit a VLM invocation. However, these policies either sample uniformly or rank windows using visual features, which ironically requires the video decoding that the budget constraints are meant to avoid. We propose audio-first triage: select windows using the lightest modality, scored before any video frame is decoded, so the approach composes naturally with token compression or quantization. The novelty lies in the objective, not the representation: rather than a per-frame sound-event detector, we train the selector to trigger once per action. This objective shift improves action coverage by 4.0-10.8 percentage points across all evaluated call rates, using frozen AudioSet-pretrained features without domain-specific sound-event labels. Using fewer than half of the available calls, the triage cuts 9-20% of VLM calls at matched coverage on EPIC-KITCHENS-100 (EK-100), surpasses uniform sampling through the mid-range on Ego4D over 247 clips, and outperforms two recent visual keyframe selectors. Code, the reference implementation and every results file this manuscript reads are at https://github.com/masjalayer/PreDecoding-AcousticTriage.

第一人称视频多模态筛选成本优化音频优先

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。