提出更真实的语音伪造定位评估方法,揭示现有模型泛化能力差的问题。
Robust Localization of Partially Fake Speech: Metrics and Out-of-Domain Evaluation
- 将定位任务视为序列异常检测,改用准确率、召回率等真实场景指标
- 模型在域内EER为7.61%,但在跨域测试中最高达43.25%
- 增加部分伪造数据可提升性能,纯真/全伪造数据反而有害
部分语音深度伪造定位面临独特挑战,相比完整语句伪造检测仍研究不足。尽管近期方法在域内表现优异,其实用性尚不明确。本文批判性分析现有评估方式的局限,特别是广泛使用的等错误率(EER)常掩盖泛化能力与部署可行性。我们建议将定位任务重新定义为序列异常检测,并倡导使用依赖阈值的指标如准确率、精确率、召回率和F1分数,以更真实反映实际表现。具体分析开源的粗到精提议修正框架(CFPRF),其在域内PartialSpoof测试集上达到20毫秒EER为7.61%,但在LlamaPartialSpoof和Half-Truth跨域测试集上分别高达43.25%和27.59%。值得注意的是,我们复现的同一模型在域内表现更差(EER 9.84%),但在跨域测试中更好(41.72%和14.98%)。这凸显了过度优化域内EER的风险,可能导致模型在真实场景中表现不佳。同时发现,增加真声或全伪造语句会降低性能,而加入部分伪造语句则能提升效果。
原文摘要 · Abstract (English)
Partial audio deepfake localization poses unique challenges and remain underexplored compared to full-utterance spoofing detection. While recent methods report strong in-domain performance, their real-world utility remains unclear. In this analysis, we critically examine the limitations of current evaluation practices, particularly the widespread use of Equal Error Rate (EER), which often obscures generalization and deployment readiness. We propose reframing the localization task as a sequential anomaly detection problem and advocate for the use of threshold-dependent metrics such as accuracy, precision, recall, and F1-score, which better reflect real-world behavior. Specifically, we analyze the performance of the open-source Coarse-to-Fine Proposal Refinement Framework (CFPRF), which achieves a 20-ms EER of 7.61% on the in-domain PartialSpoof evaluation set, but 43.25% and 27.59% on the LlamaPartialSpoof and Half-Truth out-of-domain test sets. Interestingly, our reproduced version of the same model performs worse on in-domain data (9.84%) but better on the out-of-domain sets (41.72% and 14.98%, respectively). This highlights the risks of over-optimizing for in-domain EER, which can lead to models that perform poorly in real-world scenarios. It also suggests that while deep learning models can be effective on in-domain data, they generalize poorly to out-of-domain scenarios, failing to detect novel synthetic samples and misclassifying unfamiliar bona fide audio. Finally, we observe that adding more bona fide or fully synthetic utterances to the training data often degrades performance, whereas adding partially fake utterances improves it.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。