多攻击者分阶段污染数据,可突破单阶段安全分析的盲区。
Sequential Data Poisoning in LLM Post-Training

- 设计分阶段数据污染威胁模型,模拟多个攻击者分别污染SFT与偏好数据。
- 在SFT→DPO中,分散毒预算效果优于集中;在SFT→PPO中,两类污染协同生效。
- 揭示单阶段评估的漏洞,适合关注大模型训练安全的研究者阅读。
大模型后训练包含多个阶段,如监督微调(SFT)后接基于人类反馈的强化学习(RLHF)或直接偏好优化(DPO),各阶段数据可能来自不可信来源。现有研究虽假设各阶段存在数据污染攻击,但忽略了多攻击者共谋的可能性。本文提出序列化数据污染威胁模型,即多个攻击者分别污染SFT和偏好数据集。在此模型下,我们发现‘单攻击者错觉’:单独评估时各攻击者看似无害,但跨阶段协作后真实漏洞显现。在SFT→DPO流程中,毒预算分散于各阶段的效果优于集中;在SFT→PPO流程中,仅单一阶段污染无效,二者结合却成功。结果表明,对单阶段的安全分析会系统性低估因阶段交互产生的复合漏洞。代码已开源:https://github.com/jcksanderson/sequential-poisoning。
原文摘要 · Abstract (English)
LLM post-training proceeds through multiple stages, e.g., supervised fine-tuning (SFT) followed by reinforcement learning from human feedback (RLHF) or direct preference optimization (DPO), where each stage draws data from different, potentially untrusted sources. Existing literature assumes data poisoning attacks may occur at each training stage, but neglects the possibility of multiple attackers. To study the trustworthiness of the entire post-training pipeline, we propose the threat model of sequential data poisoning, where multiple adversaries separately poison the SFT and preference datasets. Under this threat model, we identify the single-attacker illusion: each adversary, evaluated in isolation, appears to pose a negligible threat. Yet when adversaries collaborate across stages, the true vulnerability is revealed. In the SFT $\to$ DPO pipeline, their contributions are additive: splitting a fixed poison budget across stages outperforms concentrating it in either stage alone. In the SFT $\to$ PPO pipeline, their contributions are complementary: neither SFT nor reward model poisoning succeeds individually, yet their combination does. These findings show that security analyses of individual post-training stages systematically underestimate compound vulnerabilities that emerge only from their interaction. Code is available at https://github.com/jcksanderson/sequential-poisoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。