融合专家与通用模型,用标签优化解释能力,提升伪造视频检测泛化性。
Specialist-Generalist Fusion with Outcome-Supervised Rationales for Deepfake Detection

- 用冻结的专家编码器和MLLM的视觉编码器融合特征,直接生成判断结果。
- 跨域AUC从89.85提升至93.32,六组实验中五组效果提升。
- 仅用真实标签训练解释生成,推理时可关闭解释,保持原有检测精度。
通用深度伪造检测需要互补的取证与语义视觉证据。专家编码器捕捉细微篡改痕迹,但易受源域统计偏倚影响;多模态大模型(MLLM)提供更广泛的视觉-语义表征,却可能忽略细粒度取证特征。本文提出两阶段检测器:在冻结的取证编码器与MLLM原生视觉编码器之间,直接融合局部图像块特征,并由MLLM自身输出真实性判断。这种专家-通用模型对齐带来主要跨域性能提升。随后引入仅含标签的解释生成优化:模型在决策前生成自由形式的视觉解释,仅使用二分类真实标签和格式约束进行训练,无需任务特定的解释标注。推理时解释生成可选,模型仍可返回直接二分类结果。在DF40数据集上,专家-通用对齐使平均跨域AUC从89.85提升至93.32±0.44。六组对比实验中,解释优化获得93.50±0.42,其中五组优于原模型;均值差异较小,未达统计显著。在SID-Set及历史基准测试中,结合融合、输出顺序与持续训练控制实验,验证了互补视觉表征的有效性,表明仅用标签的解释优化可在不明显损害检测性能的前提下,增加可解释模式。
原文摘要 · Abstract (English)
Generalizable deepfake detection requires complementary forensic and semantic visual evidence. Specialist encoders capture subtle manipulation traces but can overfit to source-specific statistics, whereas MLLMs provide broader visual-semantic representations but may overlook fine forensic artifacts. We propose a two-stage detector in which an MLLM directly fuses patch-level features from a frozen forensic encoder with those from its native vision encoder and produces the authenticity decision itself. This specialist--generalist alignment provides the main cross-domain performance gain. We subsequently introduce outcome-supervised rationale tuning. The model generates a free-form visual rationale before its decision and is optimized using only the binary authenticity label and a format constraint, without task-specific rationale annotations. Rationale generation is optional at inference, so the tuned model can still return a direct binary score. On DF40, specialist--generalist alignment improves average cross-domain AUC from $89.85$ to $93.32\pm0.44$. Across six paired runs, rationale tuning obtains $93.50\pm0.42$ and improves five of six paired models; the mean difference is small and not statistically conclusive. Results on SID-Set and legacy benchmarks, together with fusion, output-order, and continued-training controls, demonstrate the value of complementary visual representations and show that label-only rationale tuning can add an optional explanation mode while approximately preserving direct detection performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。