arXiv:2606.22790cs.SDcs.AI2026-06

研究语音模型在算力限制下的高效扩展,给出三维度优化方案。

Scaling Audio Models Efficiently: A Joint Study of Compute Constraints and Optimization Behavior

论文配图:Scaling Audio Models Efficiently: A Joint Study of Compute Constraints and Optimization Behavior
图 1 · 摘自论文原文
  • 从模型大小、输入长度、表示分辨率三方面统一分析算力分配。
  • 发现模型越大收益越低,4秒音频对情感识别最优,降分辨率可大幅降耗。
  • 适合想高效设计语音模型的研究者和工程师参考。

本文研究自动语音识别(ASR)和语音情感识别(SER)任务中算力分配与模型性能的权衡。提出一个统一框架,分析模型规模($x_N$)、输入长度($x_T$)和表征分辨率($x_V$)三个基本算力维度。基于多模态模型的算力最优扩展进展,系统性地调整这些维度,在固定计算预算下考察其对任务性能的影响。实验基于LibriSpeech和CREMA-D数据集,揭示非线性扩展行为并识别最优操作点:(1)模型规模从Tiny(39M)增至Small(244M)使词错误率(WER)下降8.22%,而Small到Medium(769M)仅下降2.35%;(2)SER任务存在约4秒的最佳音频时长;(3)降低编码器标记分辨率可显著降低推理成本:Large-v3(1540M)在750帧时需2572 GFLOPS,1500帧时需5228 GFLOPS,WER增加不足3%。此外,基于LoRA的微调实现高效适配且性能损失极小。

原文摘要 · Abstract (English)

In this paper, we investigate the tradeoffs between compute allocation and model performance for two speech processing tasks: Automatic Speech Recognition (ASR) and Speech Emotion Recognition (SER). We propose a unified framework that analyzes three fundamental compute dimensions: model size ($x_N$), input length ($x_T$), and representation resolution ($x_V$). Motivated by recent advances in compute optimal scaling for multimodal models, we systematically vary these dimensions to examine their influence on task performance under fixed computational budgets. Our study provides insights into how compute resources can be optimally distributed across model capacity, temporal context, and representational granularity, offering practical guidelines for the design of efficient speech models. Through experiments on LibriSpeech and CREMA-D datasets, we demonstrate non-linear scaling behavior and identify optimal operating points. Our results show that (1) increasing model size yields diminishing returns: scaling Tiny (39M) to Small (244M) reduces WER by 8.22%, whereas Small to Medium (769M) reduces WER by only 2.35%; (2) an optimal audio duration of approximately 4 seconds exists for SER; and (3) reducing encoder token resolution provides an effective mechanism for lowering inference cost, Large-v3 (1540M) with 750 frames requires 2572 GFLOPS whereas with 1500 frames requires 5228 GFLOPS, with less than 3% relative increase in WER. Additionally, LoRA-based adaptation enables efficient finetuning with minimal performance degradation.

语音模型算力优化模型压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。