arXiv:2603.15685cs.MMcs.AI2026-03被引 4

用音频动态切分视频,让压缩更智能高效。

DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression

  • 以音频嵌入为锚点,通过相似度突变检测语义边界,实现动态分段。
  • 在多个数据集上保持高精度的同时,压缩率优于现有方法。
  • 无需训练,适合追求高效推理的多模态应用开发者。

多模态大模型需联合处理音视频流,但生成的长序列令牌使推理成本过高。现有压缩方法依赖固定窗口分割和基于注意力的剪枝,忽略音视频信号的分段语义结构,在激进压缩下易失效。本文提出DASH(动态音频驱动语义分块),一种无需训练的框架,将压缩与语义结构对齐。DASH以音频嵌入为语义锚点,通过余弦相似度突变检测边界候选,生成动态可变长度段落,逼近序列的分段一致组织。这些边界投影至视频令牌,作为软时间对齐的分段先验。每段内,令牌保留由三信号重要性评估器决定,融合边界结构线索、表征独特性和注意力显著性,缓解仅依赖注意力的选择稀疏偏差。该结构感知分配策略在保留关键过渡令牌的同时,减少冗余区域。在AVUT、VideoMME和WorldSense上的大量实验表明,DASH在保持竞争力或更优准确率的同时,实现了更高的压缩比。代码已开源:https://github.com/laychou666/DASH。

原文摘要 · Abstract (English)

Omnimodal large language models (OmniLLMs) jointly process audio and visual streams, but the resulting long multimodal token sequences make inference prohibitively expensive. Existing compression methods typically rely on fixed window partitioning and attention-based pruning, which overlook the piecewise semantic structure of audio-visual signals and become fragile under aggressive token reduction. We propose Dynamic Audio-driven Semantic cHunking (DASH), a training-free framework that aligns token compression with semantic structure. DASH treats audio embeddings as a semantic anchor and detects boundary candidates via cosine-similarity discontinuities, inducing dynamic, variable-length segments that approximate the underlying piecewise-coherent organization of the sequence. These boundaries are projected onto video tokens as a soft temporally co-registered segmentation prior. Within each segment, token retention is determined by a tri-signal importance estimator that fuses structural boundary cues, representational distinctiveness, and attention-based salience, mitigating the sparsity bias of attention-only selection. This structure-aware allocation preserves transition-critical tokens while reducing redundant regions. Extensive experiments on AVUT, VideoMME, and WorldSense demonstrate that DASH maintains competitive or superior accuracy while achieving higher compression ratios compared to prior methods. Code is available at: https://github.com/laychou666/DASH.

多模态令牌压缩音频驱动高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。