训练中动态调整模型结构，让参数流向更需要的地方。
Dynamic Acoustic Model Architecture Optimization in Training for ASR
- 用增删策略自动重分配参数，提升资源利用效率。
- 相同训练开销下，词错误率降低最高达6%。
- 适合想提升语音识别效果又不想增加算力的研究者。
模型架构设计复杂，现有方法或依赖人工经验，或需昂贵的神经网络搜索。本文提出DMAO框架，采用生长与删除相结合的策略，在训练过程中动态重分配参数，将资源从低效区域转移至关键部分。该方法在保持模型复杂度不变的前提下，引入极小训练开销。我们在LibriSpeech、TED-LIUM-v2和Switchboard数据集上基于CTC进行实验，结果表明：在同等训练资源下，所提方法在多种架构、模型规模和数据集上均实现相对词错误率（WER）最高6%的改进。此外，我们分析了参数重分配模式，获得若干有价值发现。
原文摘要 · Abstract (English)
Architecture design is inherently complex. Existing approaches rely on either handcrafted rules, which demand extensive empirical expertise, or automated methods like neural architecture search, which are computationally intensive. In this paper, we introduce DMAO, an architecture optimization framework that employs a grow-and-drop strategy to automatically reallocate parameters during training. This reallocation shifts resources from less-utilized areas to those parts of the model where they are most beneficial. Notably, DMAO only introduces negligible training overhead at a given model complexity. We evaluate DMAO through experiments with CTC on LibriSpeech, TED-LIUM-v2 and Switchboard datasets. The results show that, using the same amount of training resources, our proposed DMAO consistently improves WER by up to 6% relatively across various architectures, model sizes, and datasets. Furthermore, we analyze the pattern of parameter redistribution and uncover insightful findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。