arXiv:2602.00158cs.LGcs.AI2026-02

提出一种更稳定高效的语言模型概念探测方法,可低成本提取精准概念向量。

RAPTOR: Ridge-Adaptive Logistic Probes

  • 用自适应正则化逻辑回归建模,通过验证集调优正则强度获得方向稳定的向量。
  • 在多个指令微调模型上准确率媲美或超越基线,训练成本降低显著。
  • 适用于需要高效、稳定概念向量的提示工程与激活调控场景。

探测研究通过在冻结的大语言模型层表示上训练轻量级预测器,分析其编码的信息。除了分析用途外,探测器常用于探测-调节流水线:从探测器中提取概念向量,并通过加性激活调节将其注入前向传播中的层表示。该流程的有效性取决于能否获得准确、方向稳定且低成本的概念向量。为此,我们提出 RAPTOR(Ridge-Adaptive Logistic Probe),一种采用L2正则化的逻辑探测器,其验证集调优的正则强度可生成归一化权重对应的概念向量。在多个指令微调的语言模型和人工编写的概念数据集上进行的广泛实验表明,RAPTOR在准确性上达到或超过强基线,在方向稳定性方面表现良好,且训练成本大幅降低;这些定量结果得到下游调节任务的定性演示支持。最后,基于凸高斯极小极大定理(CGMT),我们在理想化的高维少量样本师生模型中对岭逻辑回归进行了机制分析,解释了正则强度如何权衡探测精度与概念向量稳定性,并得出与真实语言模型嵌入观察趋势一致的结构性预测。

原文摘要 · Abstract (English)

Probing studies what information is encoded in a frozen LLM's layer representations by training a lightweight predictor on top of them. Beyond analysis, probes are often used operationally in probe-then-steer pipelines: a learned concept vector is extracted from a probe and injected via additive activation steering by adding it to a layer representation during the forward pass. The effectiveness of this pipeline hinges on estimating concept vectors that are accurate, directionally stable under ablation, and inexpensive to obtain. Motivated by these desiderata, we propose RAPTOR (Ridge-Adaptive Logistic Probe), a simple L2-regularized logistic probe whose validation-tuned ridge strength yields concept vectors from normalized weights. Across extensive experiments on instruction-tuned LLMs and human-written concept datasets, RAPTOR matches or exceeds strong baselines in accuracy while achieving competitive directional stability and substantially lower training cost; these quantitative results are supported by qualitative downstream steering demonstrations. Finally, using the Convex Gaussian Min-max Theorem (CGMT), we provide a mechanistic characterization of ridge logistic regression in an idealized Gaussian teacher-student model in the high-dimensional few-shot regime, explaining how penalty strength mediates probe accuracy and concept-vector stability and yielding structural predictions that qualitatively align with trends observed on real LLM embeddings.

概念探测逻辑回归正则化激活调节

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。