测量语音模型实际使用的上下文长度，发现自监督模型早层用短上下文。
Effective Context in Neural Speech Models
- 提出两种测量有效上下文的方法，分析不同语音Transformer
- 自监督模型有效上下文仅在早期层增长，后期保持短上下文
- 无需修改架构即可让HuBERT流式运行，适合实时语音处理
现代神经语音模型受益于更长的上下文，尽管已有多种方法提升最大上下文长度，但很少有研究测量模型实际使用多少上下文（即有效上下文）。本文提出两种测量有效上下文的方法，并用于分析不同语音Transformer。对于有监督模型，有效上下文与任务性质相关：基频跟踪、音素分类和词分类所需的有效上下文逐级增加。对于自监督模型，有效上下文主要在早期层增长，后期保持较短，与有监督音素模型相当。由于预测时并不使用长上下文，我们证明HuBERT可在不修改架构且无需微调的情况下实现流式运行。
原文摘要 · Abstract (English)
Modern neural speech models benefit from having longer context, and many approaches have been proposed to increase the maximum context a model can use. However, few have attempted to measure how much context these models actually use, i.e., the effective context. Here, we propose two approaches to measuring the effective context, and use them to analyze different speech Transformers. For supervised models, we find that the effective context correlates well with the nature of the task, with fundamental frequency tracking, phone classification, and word classification requiring increasing amounts of effective context. For self-supervised models, we find that effective context increases mainly in the early layers, and remains relatively short -- similar to the supervised phone model. Given that these models do not use a long context during prediction, we show that HuBERT can be run in streaming mode without modification to the architecture and without further fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。