arXiv:2605.31192cs.CV2026-05

融合专家与通用模型,用标签优化解释能力,提升伪造视频检测泛化性。

Specialist-Generalist Fusion with Outcome-Supervised Rationales for Deepfake Detection

论文配图:Specialist-Generalist Fusion with Outcome-Supervised Rationales for Deepfake Detection
图 1 · 摘自论文原文
  • 用冻结的专家编码器和MLLM的视觉编码器融合特征,直接生成判断结果。
  • 跨域AUC从89.85提升至93.32,六组实验中五组效果提升。
  • 仅用真实标签训练解释生成,推理时可关闭解释,保持原有检测精度。

通用深度伪造检测需要互补的取证与语义视觉证据。专家编码器捕捉细微篡改痕迹,但易受源域统计偏倚影响;多模态大模型(MLLM)提供更广泛的视觉-语义表征,却可能忽略细粒度取证特征。本文提出两阶段检测器:在冻结的取证编码器与MLLM原生视觉编码器之间,直接融合局部图像块特征,并由MLLM自身输出真实性判断。这种专家-通用模型对齐带来主要跨域性能提升。随后引入仅含标签的解释生成优化:模型在决策前生成自由形式的视觉解释,仅使用二分类真实标签和格式约束进行训练,无需任务特定的解释标注。推理时解释生成可选,模型仍可返回直接二分类结果。在DF40数据集上,专家-通用对齐使平均跨域AUC从89.85提升至93.32±0.44。六组对比实验中,解释优化获得93.50±0.42,其中五组优于原模型;均值差异较小,未达统计显著。在SID-Set及历史基准测试中,结合融合、输出顺序与持续训练控制实验,验证了互补视觉表征的有效性,表明仅用标签的解释优化可在不明显损害检测性能的前提下,增加可解释模式。

原文摘要 · Abstract (English)

Generalizable deepfake detection requires complementary forensic and semantic visual evidence. Specialist encoders capture subtle manipulation traces but can overfit to source-specific statistics, whereas MLLMs provide broader visual-semantic representations but may overlook fine forensic artifacts. We propose a two-stage detector in which an MLLM directly fuses patch-level features from a frozen forensic encoder with those from its native vision encoder and produces the authenticity decision itself. This specialist--generalist alignment provides the main cross-domain performance gain. We subsequently introduce outcome-supervised rationale tuning. The model generates a free-form visual rationale before its decision and is optimized using only the binary authenticity label and a format constraint, without task-specific rationale annotations. Rationale generation is optional at inference, so the tuned model can still return a direct binary score. On DF40, specialist--generalist alignment improves average cross-domain AUC from $89.85$ to $93.32\pm0.44$. Across six paired runs, rationale tuning obtains $93.50\pm0.42$ and improves five of six paired models; the mean difference is small and not statistically conclusive. Results on SID-Set and legacy benchmarks, together with fusion, output-order, and continued-training controls, demonstrate the value of complementary visual representations and show that label-only rationale tuning can add an optional explanation mode while approximately preserving direct detection performance.

深度伪造可解释性多模态检测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。