用长语音段训练模型，提升语音识别与翻译中的标点和大小写准确率。
Longer is (Not Necessarily) Stronger: Punctuated Long-Sequence Training for Enhanced Speech Recognition and Translation
- 在长达60秒的完整句子上训练，使用FastConformer架构支持全注意力机制。
- 在40秒以上序列时性能趋于稳定，但比短段训练提升25%相对词错误率。
- 适合需要高精度标点和大小写的语音识别与翻译任务，代码已开源。
本文提出一种针对语音识别与翻译任务的序列到序列模型新训练方法。不同于传统在仅含小写或不完整标点和大小写（PnC）的短片段上训练，本方法采用包含完整句子且带有正确标点和大小写的长语音段进行训练。通过FastConformer架构，实现了参数量达10亿、最长支持60秒音频序列的全注意力训练。尽管加入标点和大小写能整体提升性能，但在超过40秒的长序列上，模型准确率出现平台期。所提方法在Earnings-21和Earnings-22基准上实现25%相对词错误率（WER）的显著降低，同时在多个语音识别与翻译任务中提升了整体准确性。模型权重与训练代码已通过NVIDIA NeMo开源。
原文摘要 · Abstract (English)
This paper presents a new method for training sequence-to-sequence models for speech recognition and translation tasks. Instead of the traditional approach of training models on short segments containing only lowercase or partial punctuation and capitalization (PnC) sentences, we propose training on longer utterances that include complete sentences with proper punctuation and capitalization. We achieve this by using the FastConformer architecture which allows training 1 Billion parameter models with sequences up to 60 seconds long with full attention. However, while training with PnC enhances the overall performance, we observed that accuracy plateaus when training on sequences longer than 40 seconds across various evaluation settings. Our proposed method significantly improves punctuation and capitalization accuracy, showing a 25% relative word error rate (WER) improvement on the Earnings-21 and Earnings-22 benchmarks. Additionally, training on longer audio segments increases the overall model accuracy across speech recognition and translation benchmarks. The model weights and training code are open-sourced though NVIDIA NeMo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。