通过分组调度提升音频大模型训练效率，解决多源数据冲突问题。
Heterogeneity-Aware Dataset Scheduling for Efficient Audio Large Language Model Training
- 按数据相似性分组并渐进式引入，减少梯度冲突。
- 在14个音效数据集上实现30%~40%更快收敛。
- 无需昂贵实验即可计算数据间关联，适合大规模训练。
跨多样化数据集训练通用音频大语言模型（ALLM）对实现全面音频理解至关重要，但数据异质性常导致梯度冲突和收敛缓慢。现有方法多依赖均匀混合，未充分考虑异质性管理。本文从收敛性角度分析多数据集音频问答（AudioQA）训练，提出分组顺序训练（GST）：将数据集按亲和性分组，并采用渐进调度策略引入，平衡并行训练的稳定性与串行优化的效率。为保障可扩展性，设计基于梯度的亲和性度量，无需昂贵的实证迁移性评估即可捕捉数据间关系。在涵盖语音、音乐和环境声的14个AudioQA数据集上，GST比标准并行训练快30%~40%，同时性能不低于或优于混合所有数据的训练。结果提供了理论洞见与通用高效的大规模ALLM优化框架。
原文摘要 · Abstract (English)
Training general-purpose Audio Large Language Models (ALLMs) across diverse datasets is essential for holistic audio understanding, yet it faces significant challenges due to dataset heterogeneity, which often leads to conflicting gradients and slow convergence. Despite its impact, how to explicitly manage this heterogeneity during training remains underexplored, with current practices relying primarily on uniform mixture. In this work, we analyze multi-dataset AudioQA training from a convergence perspective and propose Grouped Sequential Training (GST). GST strategically organizes datasets into affinity-aware groups and introduces them via a progressive scheduling protocol, effectively balancing the stability of parallel training with the efficiency of sequential optimization. To ensure scalability, we develop gradient-based affinity metrics that capture inter-dataset relationships without the prohibitive cost of empirical transferability estimation. Extensive evaluations on 14 AudioQA datasets spanning speech, music, and environmental sounds demonstrate that GST achieves 30--40\% faster convergence than standard parallel training while maintaining or even surpassing the performance of mix-all training. Our results provide both theoretical insights and a practical, model-agnostic framework for efficient large-scale ALLM optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。