用微调Whisper模型，边转录边检测假语音中的合成词。
Deepfake Word Detection by Next-token Prediction using Fine-tuned Whisper
- 用下一词预测机制，在转录时识别合成词
- 域内测试中检测与转录错误率均很低
- 适合低成本部署，也适用于未知生成模型
深度伪造语音可通过将真实语句中的一个或多个词替换为由语音生成模型合成的语义不同词语来实现。虽然可专门开发合成词检测器，但我们提出一种低成本方法：微调预训练的Whisper模型，通过下一词预测在转录输入语音的同时检测合成词。我们进一步探索使用部分声码化语音作为微调数据，以降低数据收集成本。实验表明，在域内测试数据上，微调后的Whisper模型具有极低的合成词检测误差率和转录误差率。在使用未见过的语音生成模型产生的合成词的域外测试数据上，其性能仍与专用的ResNet检测模型相当；然而整体性能下降表明需提升模型泛化能力。
原文摘要 · Abstract (English)
Deepfake speech utterances can be forged by replacing one or more words in a bona fide utterance with semantically different words synthesized with speech-generative models. While a dedicated synthetic word detector could be developed, we developed a cost-effective method that fine-tunes a pre-trained Whisper model to detect synthetic words while transcribing the input utterance via next-token prediction. We further investigate using partially vocoded utterances as the fine-tuning data, thus reducing the cost of data collection. Our experiments demonstrate that, on in-domain test data, the fine-tuned Whisper yields low synthetic-word detection error rates and transcription error rates. On out-of-domain test data with synthetic words produced with unseen speech-generative models, the fine-tuned Whisper remains on par with a dedicated ResNet-based detection model; however, the overall performance degradation calls for strategies to improve its generalization capability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。