用递归结构减少语音识别模型参数，性能基本不变
Rethinking Depth: A study of the Recursive-Transformer for Speech Recognition
- 同一层重复使用，通过递归压缩模型深度
- 参数量减少66%，识别准确率与原模型相当
- 适合资源受限场景的轻量化语音识别应用
基于Transformer的架构在自动语音识别（ASR）中取得显著进展，但通常伴随模型规模大幅增加。一种有前景的解决方案是通过深度递归实现层共享，即递归Transformer，其通过重复应用相同层来构建模型。尽管该方法在其他领域展现潜力，但在ASR中仍鲜少研究。本文系统研究了递归Transformer在ASR编码器中的应用，考察了递归深度及层分配对性能的影响。结果表明，递归Transformer是一种可行替代方案，尤其当递归应用于潜在空间且循环次数受限时，可在保持相近性能的同时将参数量减少66%。
原文摘要 · Abstract (English)
Transformer-based architectures have led to significant improvements in Automatic Speech Recognition (ASR), often at the cost of substantially increased model sizes. A promising approach to address this issue is layer sharing through depth recursion, commonly referred to as the Recursive-Transformer, which involves repeatedly applying the same layers within the model. Despite its potential shown in other fields, this technique remains relatively unexplored in ASR. In this paper, we present an experimental study of the Recursive-Transformer applied to ASR encoder architectures. We systematically investigate the impact of recursion depth and layer allocation within the Recursive-based Transformer. Our results demonstrate that the Recursive-Transformer is a viable alternative, especially when recurrence is applied in the latent space with a restricted number of loops, obtaining comparable performance while reducing the parameter count by 66%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。