RoPE在语音识别中表现不输传统位置编码，且训练更快。
Benchmarking Rotary Position Embeddings for Automatic Speech Recognition
- 用旋转位置编码替代传统相对位置编码，提升计算效率。
- 在100至5万小时数据上，错误率与原有方法相当或更优。
- 适合追求高效训练的语音识别研究者使用。
自注意力机制依赖位置编码来表达输入顺序。相对位置（RelPos）编码广泛用于自动语音识别（ASR），但其时间复杂度为输入长度的平方，且常与高效的GPU注意力实现不兼容。相比之下，旋转位置编码（RoPE）根据绝对位置旋转每个输入向量，时间复杂度为线性，通过自注意力点积隐式编码相对距离，通常与高效注意力兼容。然而，其在ASR中的应用仍待深入探索。本研究在涵盖读音、即兴语、清晰和嘈杂语音、不同口音的多种ASR任务上评估了RoPE，训练数据量从100小时到5万小时不等，覆盖流式与非流式场景。实验结果表明，RoPE的语音识别错误率与RelPos相当或更优，同时训练时间最多减少21%。代码已通过SpeechBrain工具包开源。
原文摘要 · Abstract (English)
Self-attention relies on positional embeddings to encode input order. Relative Position (RelPos) embeddings are widely used in Automatic Speech Recognition (ASR). However, RelPos has quadratic time complexity to input length and is often incompatible with fast GPU implementations of attention. In contrast, Rotary Positional Embedding (RoPE) rotates each input vector based on its absolute position, taking linear time to sequence length, implicitly encoding relative distances through self-attention dot products. Thus, it is usually compatible with efficient attention. However, its use in ASR remains underexplored. This work evaluates RoPE across diverse ASR tasks with training data ranging from 100 to 50,000 hours, covering various speech types (read, spontaneous, clean, noisy) and different accents in both streaming and non-streaming settings. ASR error rates are similar or better than RelPos, while training time is reduced by up to 21%. Code is available via the SpeechBrain toolkit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。