为流式语音模型设计可学习的在线寄存器，缓解未来上下文缺失问题。
Online Register for Dual-Mode Self-Supervised Speech Models: Mitigating The Lack of Future Context
- 在流式处理中添加可学习的虚拟标记，模拟未来帧信息。
- 在160毫秒分块下，LibriSpeech上相对性能提升3.4%。
- 适合低延迟语音识别场景，尤其对在线模式优化显著。
双模式自监督语音模型（S3Ms）在离线和在线模式下联合预训练，但在流式场景中因缺乏未来上下文导致注意力不匹配。为此，本文提出在线寄存器——一种附加于每个分块的可学习标记，作为未见未来帧的虚拟占位符，使模型能在不引入额外延迟的情况下补偿缺失上下文。同时，引入未来预测损失，显式引导寄存器捕捉预测性线索，增强其保留未来信息的能力。在LibriSpeech及跨域基准测试中，结果表明在线寄存器持续缩小离线与在线模式间的性能差距，在160毫秒分块设置下，LibriSpeech上实现3.4%的相对提升，尤其在低延迟场景下表现突出。
原文摘要 · Abstract (English)
Dual-mode self-supervised speech models (S3Ms), which jointly pre-trained in the offline and online mode, suffer from attention mismatch in streaming scenarios due to missing future context. To address this challenge, we proposed online registers, learnable tokens appended to each chunk in online mode. These tokens act as virtual placeholders for unseen future frames, enabling the model to compensate for missing context without introducing additional latency. Furthermore, we introduce a future prediction loss that explicitly guides the registers to capture predictive cues, thereby enriching their ability to retain future information. Experiments on LibriSpeech, and out-of-domain benchmarks demonstrate that online registers consistently reduce the performance gap between offline and online modes, achieving a 3.4% relative improvement on LibriSpeech with 160 ms chunks, especially in low-latency settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。