突破音频上下文长度限制，让大模型更好理解长音频。
Extending Audio Context for Long-Form Understanding in Large Audio-Language Models
- 仅调整音频位置编码，不改文本编码，保持原文本能力。
- 在SALMONN和Qwen2-Audio上，长音频理解性能显著提升。
- 适合需要处理超长音频的语音分析、语音助手等场景。
大型音频-语言模型（LALMs）常受制于短音频上下文窗口，即使其文本骨干支持长上下文，也限制了对长音频的理解。以往工作在单模态大模型中引入了上下文扩展方法（如YaRN），但尚未应用于LALMs。本文提出基于RoPE的局部扩展方法Partial YaRN，无需训练且模态解耦，仅修改音频标记位置，保留文本位置以维持基础模型的文本能力。进一步提出虚拟长音频训练（VLAT），通过训练时的位置增强模拟多种音频长度，使模型能泛化到训练中未见的超长输入。在SALMONN和Qwen2-Audio上的实验表明，Partial YaRN在多种设置下优于原始模型，而VLAT在未见长音频上带来显著性能提升。
原文摘要 · Abstract (English)
Large Audio-Language Models (LALMs) are often constrained by short audio context windows, even when their text backbones support long contexts, limiting long-form audio understanding. Prior work has introduced context-extension methods (e.g. YaRN) on unimodal LLMs, yet their application to LALMs remains unexplored. First, building on RoPE-based context extension, we introduce Partial YaRN, a training-free, modality-decoupled extension method that modifies only audio token positions, leaving text positions intact to preserve the base LLM's text capabilities. Second, we propose Virtual Longform Audio Training (VLAT), a training strategy that extends Partial YaRN into a training-time positional augmentation. VLAT simulates diverse audio lengths during training, enabling generalization to inputs far longer than those seen in training. Our experiments on SALMONN and Qwen2-Audio confirm that Partial YaRN outperforms the original models across wide range of settings, and VLAT provides substantial performance improvement on long audio of unseen lengths.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。