用强化学习动态调整训练样本权重,提升深度伪造检测的泛化能力。
Tutor-Student Reinforcement Learning: A Dynamic Curriculum for Robust Deepfake Detection
- Tutor-Student框架将训练过程建模为马尔可夫决策过程,动态分配样本权重。
- 通过奖励从错误到正确预测的转变,优先学习难但可学的样本,提升模型鲁棒性。
- 适合需要对抗未知伪造技术的深度伪造检测场景,尤其在数据分布变化时表现更优。
传统监督训练对所有样本赋予相同重要性,不利于学习鲁棒且可泛化的特征。本文提出一种新型的Tutor-Student强化学习(TSRL)框架,动态优化训练课程。将训练过程建模为马尔可夫决策过程,其中“导师”代理(基于近端策略优化,PPO)观察每个样本的丰富状态,包括视觉特征及其历史学习动态(如EMA损失、遗忘次数)。基于该状态,导师为样本损失分配连续权重(0-1),实现动态重加权。导师奖励依据学生(检测器)的即时性能变化,特别奖励从误判到正确预测的转变。此策略促使导师学习一个优先处理高价值样本(如难但可学样本)的课程,从而提升训练效率与效果。实验表明,该自适应课程使学生模型在未见伪造技术下的泛化能力优于传统方法。代码已开源:https://github.com/wannac1/TSRL。
原文摘要 · Abstract (English)
Standard supervised training for deepfake detection treats all samples with uniform importance, which can be suboptimal for learning robust and generalizable features. In this work, we propose a novel Tutor-Student Reinforcement Learning (TSRL) framework to dynamically optimize the training curriculum. Our method models the training process as a Markov Decision Process where a ``Tutor'' agent learns to guide a ``Student'' (the deepfake detector). The Tutor, implemented as a Proximal Policy Optimization (PPO) agent, observes a rich state representation for each training sample, encapsulating not only its visual features but also its historical learning dynamics, such as EMA loss and forgetting counts. Based on this state, the Tutor takes an action by assigning a continuous weight (0-1) to the sample's loss, thereby dynamically re-weighting the training batch. The Tutor is rewarded based on the Student's immediate performance change, specifically rewarding transitions from incorrect to correct predictions. This strategy encourages the Tutor to learn a curriculum that prioritizes high-value samples, such as hard-but-learnable examples, leading to a more efficient and effective training process. We demonstrate that this adaptive curriculum improves the Student's generalization capabilities against unseen manipulation techniques compared to traditional training methods. Code is available at https://github.com/wannac1/TSRL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。