用kNN提升Whisper语音识别，不训练模型也能适应不同口音和说话人。
kNN For Whisper And Its Effect On Bias And Speaker Adaptation
- 在推理时通过外部数据存储进行词元级kNN搜索，无需微调模型。
- 对不同性别、口音和年龄的说话人识别准确率均有提升，尤其改善了弱口音表现。
- 适合需要快速适配新说话人或避免灾难性遗忘的语音应用开发者。
语音识别性能受语言、领域及说话人特征（如口音）影响，但针对这些类别微调模型可能导致灾难性遗忘。词元级k近邻搜索（kNN）是一种非参数方法，最初用于自然语言生成与机器翻译，在推理阶段通过外部数据存储进行搜索，无需训练底层模型。本文证明Whisper——一种基于Transformer的端到端语音模型——可从中获益。我们对比了语音与文本场景下的差异，探讨了其在说话人适配中的应用，并分析了按性别、口音和年龄划分的改进效果。
原文摘要 · Abstract (English)
Speech recognition performance varies by language, domain, and speaker characteristics such as accent, but fine-tuning a model on any of these categories may lead to catastrophic forgetting. Token-level $k$ nearest neighbor search ($k$NN), first proposed for neural sequence decoders for natural language generation (NLG) and machine translation (MT), is a non-parametric method that instead adapts using inference-time search in an external datastore, without training the underlying model. We show that Whisper, a transformer end-to-end speech model, benefits from $k$NN. We investigate the differences between the speech and text setups. We discuss implications for speaker adaptation, and analyze improvements by gender, accent, and age.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。