大模型能察觉并识别激活值中的噪声和丢弃扰动
Language models recognize dropout and Gaussian noise applied to their activations
- 通过掩码或高斯噪声扰动激活值，测试模型检测能力
- 8B至32B规模模型均实现近乎完美的扰动定位与识别
- 展现对训练机制的隐式感知，或具安全警示意义
我们提供证据表明，语言模型能够检测、定位并部分描述施加于其激活值上的扰动差异。具体而言，我们在目标句上施加两种扰动：(a) 掩码激活值（模拟丢弃），或 (b) 添加高斯噪声。随后提出多选问题，如“哪一句被扰动了？”或“哪种扰动被应用了？”。我们在 Llama、Olmo 和 Qwen 系列模型（8B 到 32B 规模）上进行测试，所有模型均能轻松检测并定位扰动，常达完美准确率。当在上下文中提示时，这些模型还能学会区分丢弃与高斯噪声。值得注意的是，Qwen3-32B 在零样本条件下识别扰动类型的能力随扰动强度提升而增强，若上下文标签被反转，准确率则下降，暗示其存在对正确标签的先验偏好——即使在控制条件下亦然。由于丢弃是训练正则化手段，而高斯噪声常用于推理阶段，这一现象可能暗示一种不依赖数据的“训练意识”信号，对人工智能安全具有重要意义。
原文摘要 · Abstract (English)
We provide evidence that language models can detect, localize and, to a certain degree, verbalize the difference between perturbations applied to their activations. More precisely, we either (a) mask activations, simulating dropout, or (b) add Gaussian noise to them, at a target sentence. We then ask a multiple-choice question such as "Which of the previous sentences was perturbed?" or "Which of the two perturbations was applied?". We test models from the Llama, Olmo, and Qwen families, with sizes between 8B and 32B, all of which can easily detect and localize the perturbations, often with perfect accuracy. These models can also learn, when taught in context, to distinguish between dropout and Gaussian noise. Notably, Qwen3-32B's zero-shot accuracy in identifying which perturbation was applied improves as a function of the perturbation strength and, moreover, decreases if the in-context labels are flipped, suggesting a prior for the correct ones -- even modulo controls. Because dropout has been used as a training-regularization technique, while Gaussian noise is sometimes added during inference, we discuss the possibility of a data-agnostic "training awareness" signal and the implications for AI safety.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。