arXiv:2606.29604cs.LGcs.AI2026-06被引 14

用少量数据发现大模型隐藏行为,提升安全性和对齐性。

Mechanistically Eliciting Latent Behaviors in Language Models

论文配图:Mechanistically Eliciting Latent Behaviors in Language Models
图 1 · 摘自论文原文
  • 通过无监督张量分解法寻找可解释的低秩适配器,激活模型潜藏行为
  • 仅用单个示例就成功触发复杂多标记行为,性能接近有监督方法
  • 可暴露模型隐藏缺陷并防止对齐伪装,适合模型安全评估与对齐研究

我们旨在发现能够激发语言模型内部隐藏行为模式的多样化、泛化性强的扰动。这类扰动有助于重塑模型行为并系统评估潜在风险。本文提出因果扰动激发(CPE),一种无监督方法,可发现可解释的低秩适配器(LoRAs),以激发这些隐含行为。CPE 使用基于启发式的张量分解算法,对深层Transformer层的计算进行分解。该方法表现出极高的数据效率,仅需一个样本即可学习大量可解释的LoRAs。尽管是无监督方法,其在某些情况下仍能与有监督的暴力枚举方法相媲美——例如,在Countdown任务中,CPE与匹配运行时间的GRPO在Qwen3-8B上表现相当(85% vs 87%),证明其能高效激发复杂多标记行为。由于无需标注数据,CPE还能揭示隐藏失败模式,如“拖延”行为,在Taylor等人(2025)引入密码锁的Llama3-70B版本上,恢复了85%被锁定的BigCodeBench性能。此外,由于在权重空间而非标记空间探索行为,CPE可能缓解探索劫持(exploration hacking)这一由高度自知智能体引发的对齐失败问题(Ngo, 2022)。事实上,我们发现CPE几乎消除了Hughes等人(2025)开发的基于Llama3-70B的模型生物体中的对齐伪装行为(Greenblatt等,2024)。最后,我们将CPE用于初始化GPT-OSS-20B,在易发生奖励劫持的环境中运行GRPO时进入对齐盆地。CPE为系统探索模型潜藏行为提供了高效工具,显著增强了人工智能系统的对齐与安全性评估能力。

原文摘要 · Abstract (English)

We aim to discover diverse, generalizable perturbations of LLM internals that can surface hidden behavioral modes. Such perturbations could help reshape model behavior and systematically evaluate potential risks. We introduce Causal Perturbative Elicitation (CPE), an unsupervised method for discovering interpretable low-rank adapters (LoRAs) that can elicit these latent behaviors. CPE decomposes the computations of a deep transformer slice using a heuristic tensor-decomposition-based algorithm. CPE exhibits remarkable data efficiency, learning a large number of interpretable LoRAs from a single example. Even though CPE is unsupervised, we find that in some cases it can be competitive with supervised elicitation methods via brute-force enumerative search over weight space. For instance, CPE performs similarly to matched-wall-clock-time GRPO on the Countdown task for Qwen3-8B (85% vs 87%), demonstrating that CPE can efficiently elicit complex multi-token behaviors. Since CPE is unsupervised, it can also surface hidden failure modes, such as sandbagging, restoring 85% of locked BigCodeBench performance on a password-locked version of Llama3-70B introduced by Taylor et al. (2025). Additionally, since CPE explores behaviors in weight-space rather than token-space it can potentially ameliorate exploration hacking, a misalignment failure which may arise in sufficiently self-aware AI models (Ngo, 2022). In fact, we find that CPE virtually eliminates alignment-faking (Greenblatt et al., 2024) behavior in a Llama3-70B-based model organism developed by Hughes et al. (2025). Finally, we find that CPE can be used to initialize GPT-OSS-20B in an aligned basin when running GRPO on an environment prone to reward-hacking. By providing a data-efficient method to systematically explore the space of latent model behaviors, CPE yields a powerful tool for aligning AI systems and evaluating their safety.

模型对齐行为挖掘安全评估低秩适配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。