攻击语言模型的隐式推理过程,让模型答错却无迹可寻。
Thinking Wrong in Silence: Backdoor Attacks on Continuous Latent Reasoning
- 通过扰动输入嵌入向量,利用多轮推理放大攻击信号。
- 在多个模型和任务上攻击成功率超99%,且不影响正常性能。
- 揭示了连续推理中的新漏洞,适合研究安全与可解释性者阅读。
新一代语言模型完全在连续隐藏状态中进行推理,不生成任何词元,不留审计痕迹。我们发现这种沉默创造了全新的攻击面。ThoughtSteer 在输入层扰动单一嵌入向量;模型自身的多轮推理将该扰动放大为被劫持的潜在轨迹,可靠地产生攻击者指定的答案,同时对所有基于词元的防御手段保持结构不可见。在两种架构(Coconut 和 SimCoT)、三个推理基准以及参数量从124M到3B的模型上,ThoughtSteer 攻击成功率≥99%,接近基线的清洁准确率,无需重训练即可迁移至未见基准(94–100%),成功规避所有五种评估的主动防御,并经受住25轮干净微调。我们追溯其成因于统一机制:潜在空间中的神经坍缩将触发表示拉向紧密几何吸引子,解释了为何防御失效,也说明有效后门必须留下线性可分签名(探测AUC≥0.999)。然而一个惊人悖论浮现:即使模型输出错误答案,单个潜在向量仍编码正确答案。对抗信息不在任一向量中,而在整体轨迹中,确立后门扰动为连续推理机制可解释性的新视角。代码与检查点已公开。
原文摘要 · Abstract (English)
A new generation of language models reasons entirely in continuous hidden states, producing no tokens and leaving no audit trail. We show that this silence creates a fundamentally new attack surface. ThoughtSteer perturbs a single embedding vector at the input layer; the model's own multi-pass reasoning amplifies this perturbation into a hijacked latent trajectory that reliably produces the attacker's chosen answer, while remaining structurally invisible to every token-level defense. Across two architectures (Coconut and SimCoT), three reasoning benchmarks, and model scales from 124M to 3B parameters, ThoughtSteer achieves >=99% attack success rate with near-baseline clean accuracy, transfers to held-out benchmarks without retraining (94-100%), evades all five evaluated active defenses, and survives 25 epochs of clean fine-tuning. We trace these results to a unifying mechanism: Neural Collapse in the latent space pulls triggered representations onto a tight geometric attractor, explaining both why defenses fail and why any effective backdoor must leave a linearly separable signature (probe AUC>=0.999). Yet a striking paradox emerges: individual latent vectors still encode the correct answer even as the model outputs the wrong one. The adversarial information is not in any single vector but in the collective trajectory, establishing backdoor perturbations as a new lens for mechanistic interpretability of continuous reasoning. Code and checkpoints are available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。