提出局部合并方法,让音频压缩更智能且保留上下文。
Locality Matters for Training-Free Audio Token Compression in Audio-Language Models

- 在时间窗内合并相似音频片段,保持上下文连续性。
- 局部合并比全局匹配更适合生成式任务,尤其强压缩下提升明显。
- 无需训练,适配多模型,特别适合语音描述类应用。
音频-语言模型(ALMs)在音频描述、问答和开放域音频理解中应用广泛,但长音频前缀序列导致推理成本高,占用上下文预算、增加内存消耗,在资源受限或低延迟场景下部署困难。现有无训练音频令牌压缩方法主要依赖固定池化或基于分数的剪枝:前者无视内容,后者虽能保留显著片段却丢弃邻近声学上下文。本文提出局部时序二分合并(LTBM),一种无训练的编码空间压缩方法,通过显式时间窗口约束合并相邻相似音频令牌。此外,通过控制全局合并变体,验证了时间局部性是否为有效归纳偏置。在AudioCaps、Clotho和MMAU数据集上使用Qwen2-Audio的实验表明,局部性对任务有影响:在多种压缩设置下,尤其强压缩时,局部感知合并更利于生成式任务;而全局匹配在多项选择型音频理解任务中表现更优。在Audio Flamingo 3上的跨骨干验证进一步支持局部合并对生成任务的优势,尤其在中等及激进压缩下。
原文摘要 · Abstract (English)
Audio-language models (ALMs) are increasingly used for audio captioning, question answering, and open-ended audio understanding, but their inference cost remains high when audio inputs are represented as long prefix-token sequences. These audio prefixes consume context budget, increase memory usage, and make deployment harder in resource-constrained or latency-sensitive settings. Existing training-free audio-token reduction methods mainly rely on fixed pooling or score-based pruning. Fixed pooling is content-agnostic, while score-based pruning can preserve isolated salient tokens but discard nearby acoustic context. We propose Local Temporal Bipartite Merging (LTBM), a training-free encoder-space compression method that merges similar nearby audio tokens under an explicit temporal window constraint. Beyond introducing LTBM, we use a controlled Global Merge variant to isolate whether temporal locality itself is a useful inductive bias for audio-token compression. Experiments on AudioCaps, Clotho, and MMAU with Qwen2-Audio show evidence of a task-dependent locality effect: locality-aware merging is more favorable for captioning at several compression settings, especially under stronger compression, while global matching is more competitive for multiple-choice audio understanding. A cross-backbone validation on Audio Flamingo 3 further supports the captioning-side advantage of locality-aware merging under moderate and aggressive compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。