让语音模型按需输出不同文字，零样本实现语言转写
Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliteration
- 发现语音模型激活空间中脚本信息线性编码
- 推理时调整激活值可控制输出脚本，支持非常规配对
- 无需重新训练，适配所有Whisper模型规模
多语言语音基础模型（如Whisper）在大规模网络数据上训练，每种语言包含多种方言变体。不同方言常使用不同书写系统表达同一语言，导致语音识别输出的书写形式具有不确定性。我们发现，脚本信息在线性编码于多语言语音模型的激活空间中，通过在推理时修改激活值，可直接控制输出脚本。实验表明，在测试时添加特定脚本向量，即使在非典型语言-脚本组合（如意大利语用西里尔字母、日语用拉丁字母）下也能实现脚本转换。该方法应用于语音识别输出的后处理脚本控制，在所有尺寸的Whisper模型上均取得竞争性性能。
原文摘要 · Abstract (English)
Multilingual speech foundation models such as Whisper are trained on web-scale data, where data for each language consists of a myriad of regional varieties. However, different regional varieties often employ different scripts to write the same language, rendering speech recognition output also subject to non-determinism in the output script. To mitigate this problem, we show that script is linearly encoded in the activation space of multilingual speech models, and that modifying activations at inference time enables direct control over output script. We find the addition of such script vectors to activations at test time can induce script change even in unconventional language-script pairings (e.g. Italian in Cyrillic and Japanese in Latin script). We apply this approach to inducing post-hoc control over the script of speech recognition output, where we observe competitive performance across all model sizes of Whisper.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。