用合成音频信号精准提升大模型听觉理解能力
SpectCount: Spectrotemporal Counting via Synthetic Signals Improves Large Audio Language Models
- 基于实时生成的合成音频，针对性优化听觉感知弱点
- 在多个音效、音乐、语音任务上显著提效，无需真实标注数据
- 适合追求高效训练与强听觉理解的AI研究者
大音频语言模型（LALMs）通过音频编码器和大规模音频数据扩展了大语言模型。然而，高质量标注音频数据的稀缺仍是制约其发展的根本瓶颈。通过探测信号可检测性分析，我们识别出基础LALM在精细频时感知上的薄弱环节。为此，我们提出频时计数（SpectCount）方法，一种基于实时生成的完全合成音频信号的数据高效微调方案，不依赖真实音频、标注或预训练生成模型。SpectCount不仅解决了上述缺陷，还在多个涵盖声音、音乐和语音的听觉基准测试中实现了性能提升，这些任务均未在微调阶段出现。结果表明，针对弱点设计的合成信号为提升LALMs的听觉理解能力提供了一条数据高效的路径。
原文摘要 · Abstract (English)
Large audio language models (LALMs) extend large language models with an audio encoder and large-scale audio data. However, the scarcity of high-quality annotated audio data remains a fundamental bottleneck for scaling. Through probing signal detectability analysis, we identify fine-grained spectrotemporal perceptual weaknesses in a foundation LALM. To address these challenges, we propose Spectrotemporal Counting (SpectCount), a data-efficient fine-tuning approach based on fully synthetic audio signals generated on-the-fly, without relying on real-world audio, annotations, or pretrained generative models. SpectCount not only resolves the observed weaknesses but also improves performance on diverse auditory benchmarks spanning sound, music, and speech, unseen during fine-tuning. These results suggest that weakness-targeted synthetic signals provide a data-efficient path toward enhanced auditory understanding capabilities in LALMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。