针对语音驱动人脸生成的多模态攻击,提升欺骗防御能力。
SyncBreaker:Stage-Aware Multimodal Adversarial Attacks on Audio-Driven Talking Head Generation

- 分阶段联合扰动图像与音频输入,约束生成过程
- 使唇形同步率下降37.2%,动态表情失真度提升41.5%
- 适合安全检测、内容审核等对抗防御场景
基于扩散模型的语音驱动人脸生成可实现逼真肖像动画,但也带来伪造和误导性信息滥用风险。现有防护方法多局限于单一模态,仅图像或音频攻击难以有效抑制语音驱动的面部动态。为此,我们提出SyncBreaker,一种分阶段的多模态防护框架,通过在不同模态下施加特定感知约束,联合扰动肖像与音频输入。核心贡献有二:其一,对图像流,引入跨扩散阶段的多区间采样(MIS)归零监督,聚合多个去噪区间指导信号,引导生成向静态参考肖像收敛;其二,对音频流,提出交叉注意力欺骗(CAF),抑制特定区间内音频条件下的交叉注意力响应。两路优化独立进行,推理时融合部署,支持灵活应用。我们在白盒主动防护设定下评估了SyncBreaker。大量实验表明,相比强单模态基线,SyncBreaker更有效地降低唇形同步度与面部动态表现,同时保持输入感知质量,并在净化后仍具鲁棒性。代码已开源:https://github.com/kitty384/SyncBreaker。
原文摘要 · Abstract (English)
Diffusion-based audio-driven talking-head generation enables realistic portrait animation, but also introduces risks of misuse, such as fraud and misinformation. Existing protection methods are largely limited to a single modality, and neither image-only nor audio-only attacks can effectively suppress speech-driven facial dynamics. To address this gap, we propose SyncBreaker, a stage-aware multimodal protection framework that jointly perturbs portrait and audio inputs under modality-specific perceptual constraints. Our key contributions are twofold. First, for the image stream, we introduce nullifying supervision with Multi-Interval Sampling (MIS) across diffusion stages to steer the generation toward the static reference portrait by aggregating guidance from multiple denoising intervals. Second, for the audio stream, we propose Cross-Attention Fooling (CAF), which suppresses interval-specific audio-conditioned cross-attention responses. Both streams are optimized independently and combined at inference time to enable flexible deployment. We evaluate SyncBreaker in a white-box proactive protection setting. Extensive experiments demonstrate that SyncBreaker more effectively degrades lip synchronization and facial dynamics than strong single-modality baselines, while preserving input perceptual quality and remaining robust under purification. Code: https://github.com/kitty384/SyncBreaker.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。