用文本化声学特征提升大模型对未知伪造语音的检测能力
Textual Acoustic Grounding for Generalizable LLM-Based Deepfake Voice Detection
- 将openSMILE提取的声学特征转为文本提示，桥接音频与语言模型的模态鸿沟
- 在ITW和MLAAD数据集上实现超16.2%的宏平均F1提升，跨域性能最优
- 冻结LLM+文本化声学提示，资源消耗低且效果优于微调
伪造语音检测在未见领域上泛化能力差。尽管音频大语言模型（ALLMs）展现潜力，但连续音频嵌入所捕捉的细微声学特征与语言模型语义空间之间存在显著模态差距，这一关键瓶颈尚未被充分探索。本文通过评估多种音频编码器与Qwen LLM（0.5B至7B参数）的集成效果，发现仅微调LLM会导致跨域过拟合，而冻结LLM反而是更优、更高效的基线。为此，提出一种跨模态提示策略，将基于语言知识驱动的声学特征（通过openSMILE提取）作为结构化文本标记注入模型。该显式文本化声学接地不仅增强冻结基线性能，还使后续微调更有效。最终方法在跨域ITW和MLAAD基准上达到当前最优表现，相比现有ALLM基线，宏平均F1提升超过16.2%，同时保持良好的域内性能。本文所有模型均公开可用。
原文摘要 · Abstract (English)
Deepfake voice detection suffers from poor generalization across unseen domains. While Audio Large Language Models (ALLMs) show promise, the modality gap between continuous audio embeddings which capture the subtle acoustic details necessary for deepfake detection and the semantic space of LLMs remains a critical, underexplored bottleneck. We address this by benchmarking diverse audio encoders integrated with Qwen LLMs (0.5B to 7B parameters). First, we demonstrate that fine-tuning the LLM alone risks out-of-domain overfitting, making a frozen LLM a stronger, resource-efficient baseline. Second, to explicitly bridge the modality gap, we introduce a cross-modal prompting strategy that injects linguistic-knowledge-driven acoustic features (via openSMILE) as structured text tokens. This explicit textual grounding not only enhances the frozen baseline but also makes LLM fine-tuning more effective. Ultimately, our approach demonstrates state-of-the-art resilience on the out-of-domain ITW and MLAAD benchmarks, yielding over \textbf{16.2\%} absolute improvement in Macro-F1 over existing ALLM baselines while maintaining competitive in-domain performance. All models reported in this work are \href{https://huggingface.co/01Yassine/AudioLLM-Deepfake-Detection}{publicly available}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。