通过检索式压缩技术,提升大模型对一小时级音视频的长时理解能力。
AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression

- 将多模态压缩重构为顶K检索问题,动态筛选关键帧与音频片段。
- 在长视频基准上分别领先第二名4.9和5.5分,准确率显著提升。
- 适用于需要长时上下文理解的场景,如会议记录、纪录片分析。
多模态大模型在短时音视频理解上取得显著进展,但长时音视频理解仍受限于上下文窗口大小与严重的信息冗余。为此,我们提出AVOC框架,用于增强全模态大模型对长时音视频的理解能力。AVOC在模态编码器与大模型主干之间引入可学习的令牌压缩模块,将多模态令牌压缩重新构想为一个顶-K检索问题:在固定上下文预算下,模块需检索出最能支持回答用户查询的紧凑令牌子集。我们借鉴信息检索中的三个经典标准——相关性、重要性与多样性——设计了适配音视频理解的专用机制,并集成至统一的检索式压缩流水线中。实验表明,AVOC在长时音视频基准OmniVideoBench和LVOmniBench上分别以平均准确率超越第二名4.9和5.5个百分点,且在长达一小时的音视频找针任务中保持稳定表现。
原文摘要 · Abstract (English)
Multimodal Large Language Models have achieved remarkable progress in short-form audio-video understanding, yet long-form audio-video comprehension remains challenged by limited context windows and severe information redundancy. To address these bottlenecks, we propose AVOC, a framework for long-form audio-video understanding in Omni-modal Large Language Models. AVOC introduces a learnable token compression module between the modality encoders and the LLM backbone. We reframe multimodal token compression as a top-$K$ retrieval problem: given a fixed context budget, the module must retrieve a compact subset of tokens that best supports answering the user query. We draw inspiration from three classical Information Retrieval criteria for selecting informative units from a large candidate pool: relevance, importance, and diversity. AVOC instantiates each criterion as a tailored mechanism for audio-video understanding, and integrates them into a unified retrieval-style compression pipeline. Experiments show that AVOC achieves state-of-the-art performance on long-form audio-video benchmarks, surpassing the second-best model by 4.9 and 5.5 points in average accuracy on OmniVideoBench and LVOmniBench, respectively. Moreover, AVOC maintains robust performance on Audio-Video Needle-in-a-Haystack task at durations up to one hour.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。