构建去重增强的音频推理数据集,提升大模型后训练效果。
AudioDER: A Deduplication-Enhanced Reasoning Dataset for Post-Training Large Audio-Language Models

- 通过三阶段流程过滤声学冗余,生成高质量多选问答数据。
- 建成28.6万条样本的AudioRE数据集,性能优于更大但冗余的数据。
- 适合研究音频语言模型后训练与数据质量优化的研究者。
近期预训练大型音频-语言模型(LALMs)在语音、声音和音乐任务中展现出强大能力。为避免从头预训练的高昂成本,后训练已成为主流范式。然而,后训练效果高度依赖训练语料质量。我们发现,现有后训练语料常由多个公共音频数据集拼接而成,因来源平台重叠导致大量声学冗余,重复暴露于相似音频模式中,即使数据量增加也难带来性能提升。为此,我们提出三阶段数据构建流程:先进行声学冗余过滤,再将保留样本转化为统一的带思维链的多选问答格式,最后实施质量验证与筛选。基于此流程,我们构建了包含约28.6万实例的AudioRE数据集,涵盖声音、语音与音乐。在AudioRE上进行监督微调,显著提升Qwen2-Audio-7B-Instruct在多样化音频理解与推理基准上的表现,优于使用更大数据量但未去冗余原始语料训练的模型。结果验证了去冗余数据构建流程的有效性及AudioRE数据集的价值,进一步强调了降低音频-语言后训练中声学冗余的重要性。为促进后续研究,我们将公开AudioRE数据集及对应的微调后Qwen2-AudioRE检查点。
原文摘要 · Abstract (English)
Recent advances in pretrained large audio-language models (LALMs) have demonstrated strong capabilities across speech, sound, and music. To adapt these models to downstream tasks without the cost of pretraining from scratch, post-training has become a widely adopted paradigm. However, the effectiveness of post-training depends critically on the quality of the training corpus. We observe that existing post-training corpora, often constructed by aggregating public audio datasets, suffer from substantial acoustic redundancy, as many of these datasets are sourced from overlapping media platforms. Such redundancy leads to repeated exposure to similar acoustic patterns, causing diminishing returns in performance despite increased data volume. address this issue, we propose a three-stage data construction pipeline that performs acoustic redundancy filtering, converts retained samples into a unified multiple-choice question-answering format with chain-of-thought generation, and finally applies quality verification and filtering. Using this pipeline, we construct AudioRE, a post-training dataset of approximately 286k instances spanning sound, speech, and music. Supervised fine-tuning on AudioRE consistently improves the performance of Qwen2-Audio-7B-Instruct across diverse audio understanding and reasoning benchmarks, outperforming models trained on the unfiltered raw corpus with substantially more instances. These results validate the effectiveness of our redundancy-aware data construction pipeline and the resulting AudioRE dataset, and further highlight the importance of minimizing acoustic redundancy in audio-language post-training. To facilitate future research, we will release both the AudioRE and the fine-tuned Qwen2-AudioRE checkpoint.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。