用多模态大模型检测语音伪造，效果依赖特定训练。
Investigating the Viability of Employing Multi-modal Large Language Models in the Context of Audio Deepfake Detection
- 用音频+多轮问答提示，让模型学习跨模态特征。
- 微调后在同领域数据上表现良好，零样本效果差。
- 适合需要少量标注的语音伪造检测场景。
尽管视觉语言模型和多模态大语言模型在图像与视频伪造检测中展现出强大泛化能力，但其在语音伪造检测中的应用仍鲜有探索。本文旨在考察多模态大语言模型（MLLMs）在语音伪造检测中的潜力。通过将音频输入与多种文本提示结合，以问答形式进行二分类判断，探索模型是否能学习到跨模态的鲁棒表示。我们假设，这种基于特征引导的推理机制有助于深化多模态理解，促进稳健特征学习。评估了 Qwen2-Audio-7B-Instruct 与 SALMONN 两种 MLLM 模型，在零样本和微调两种模式下的性能。实验表明，结合多提示策略可为语音伪造检测提供可行路径。模型在未经过任务特训时表现不佳，且难以泛化至域外数据；但在少量监督下，于同域数据上表现良好，显示出在语音伪造检测中的潜在价值。
原文摘要 · Abstract (English)
While Vision-Language Models (VLMs) and Multimodal Large Language Models (MLLMs) have shown strong generalisation in detecting image and video deepfakes, their use for audio deepfake detection remains largely unexplored. In this work, we aim to explore the potential of MLLMs for audio deepfake detection. Combining audio inputs with a range of text prompts as queries to find out the viability of MLLMs to learn robust representations across modalities for audio deepfake detection. Therefore, we attempt to explore text-aware and context-rich, question-answer based prompts with binary decisions. We hypothesise that such a feature-guided reasoning will help in facilitating deeper multimodal understanding and enable robust feature learning for audio deepfake detection. We evaluate the performance of two MLLMs, Qwen2-Audio-7B-Instruct and SALMONN, in two evaluation modes: (a) zero-shot and (b) fine-tuned. Our experiments demonstrate that combining audio with a multi-prompt approach could be a viable way forward for audio deepfake detection. Our experiments show that the models perform poorly without task-specific training and struggle to generalise to out-of-domain data. However, they achieve good performance on in-domain data with minimal supervision, indicating promising potential for audio deepfake detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。