攻击者仅控制重放索引,就能悄悄降低持续学习模型性能。
Amnesia: A Stealthy Replay Attack on Continual Learning Dreams

- 通过调整重放类别分布,在不改变重放率的前提下制造干扰。
- 在多个基准上使最终准确率下降,后向迁移能力显著恶化。
- 攻击隐蔽性强,尤其KL变体难以被批量或滚动审计发现。
持续学习模型常使用经验重放缓解灾难性遗忘,但其对重放采样干扰的鲁棒性尚未深入研究。现有攻击多修改输入或训练流程(如投毒、后门),缺乏可审计性约束,现实性不足。可审计性指监控器可通过可见日志(如重放索引/标签统计)验证采样合规性:实际重放类别直方图应接近基线p0,且每批次/滑动窗口内重放率不变。本文研究仅能控制重放索引选择的有限权限内部人员,保持在队列优先级等可审计范围内。提出Amnesia攻击:在两个预算下最大化破坏力——可视性预算δ限制真实分布与基线p0的总变差(TV)或KL散度;质量预算f固定重放率。攻击分两步:(i) 计算轻量级类别效用(如指数移动平均损失或置信度),将基线分布倾斜至有害类别;(ii) 使用高效优化器(KL散度用指数倾斜,TV用平衡质量再分配)将倾斜投影回δ球内。滑动窗口调度器实现滚动审计。在多个挑战性CL基准和强重放基线上,Amnesia始终降低最终准确率(ACC)并恶化后向迁移(-BWT)。KL变体影响大且难检测,即使在多种审计方案下仍隐蔽;TV变体更致命但易被细粒度分类约束察觉。结果揭示仅控制重放索引即构成实际可审计威胁面,并建立影响与可见性的权衡准则。
原文摘要 · Abstract (English)
Continual learning (CL) models often use experience replay to reduce catastrophic forgetting, but their robustness to replay sampling interference remains underexplored. Existing CL attacks alter inputs or training pipelines (poisoning/backdoors) and rarely include explicit auditable constraints, limiting realism. Here, auditability means a monitor can verify compliance from sampler-visible telemetry - e.g., logged replay index/label statistics - by checking that the realized replay class histogram stays close to a nominal baseline and that replay rate is unchanged per batch and/or over a rolling window. We study a limited-privilege insider who controls only replay index selection, not pixels, labels, or model parameters, while staying within auditable limits such as queue priorities. We introduce Amnesia, a replay composition attack that maximizes degradation under two budgets: a visibility budget delta bounding the TV/KL divergence from a nominal class histogram p0, and a mass budget f fixing the replay rate. Amnesia has two steps: (i) compute lightweight class utilities, such as EMA loss or confidence, to tilt p0 toward harmful classes; and (ii) project the tilt back into the delta-ball using efficient KL (exponential tilt) or TV (balanced mass redistribution) optimizers. A windowed scheduler enforces rolling audits. Across challenging CL benchmarks and strong replay baselines, Amnesia consistently lowers final accuracy (ACC) and worsens backward transfer (-BWT). The KL variant delivers high impact while remaining largely undetected under multiple audit schemes, including per-batch and rolling-window checks. The TV variant is more damaging but easier to detect, especially under tight per-class constraints. These results expose index-only replay control as a practical, auditable threat surface in CL systems and establish a principled impact-visibility trade-off.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。