发现语言模型中正确性信息的几何结构，可精准定位并操控错误输出。
The Confidence Manifold: Geometric Structure of Correctness Representations in Language Models
- 在残差流中识别出2-8维子空间内的正确性信号，仅需25个样本即可恢复90%准确率。
- 操控信号可使幻觉率变化达9.1个百分点，擦除后检测能力归零。
- 适合研究模型内部表征、可信推理与幻觉控制的开发者或研究人员。
当语言模型声称‘澳大利亚首都是悉尼’时，它是否知道这是错的？模型对事实和谬误的表达同样流畅，因此无法通过输出不确定性判断。尽管已知真理相关信号存在于残差流中，但其几何结构仍不明确：信号占据多少维度、检测器有多简单、是否可迁移。本文在11个模型（124M-14B）中刻画该结构，并通过激活引导、概念擦除与分布式对齐搜索进行因果验证。结果表明，信号结构简单：二维至八维子空间中的两个类别中心可匹配训练线性探测器；仅25个标注样本即可在GPT-2上恢复90%全数据AUC。引导操作使六个模型的幻觉率变化达9.1点，擦除后检测能力降至随机水平；唯一能限制秩的分布式对齐搜索表明，最多存在五个因果维度。内部优势具有场景特异性：在对抗性谬误上，探测器显著优于P(True)与语义熵，但在标准问答任务上表现相当。单数据集探测器迁移近似随机，而联合多数据集训练后恢复0.73-0.91 AUC。类别中心距离与探测性能一致，表明检测本质是均值偏移，而非学习所得。
原文摘要 · Abstract (English)
When a language model asserts that "the capital of Australia is Sydney," does it know this is wrong? Models assert misconceptions with the same fluency as facts, so the question cannot be answered from output uncertainty. Truth-related signals are known to exist in the residual stream, but not their geometry: how many dimensions carry the signal, how simple a detector can be, and whether it transfers. We characterize this geometry across 11 models (124M-14B) and test it causally with activation steering, concept erasure, and distributed alignment search. The structure is simple: two class centroids in a 2-8 dimensional subspace match a trained linear probe, and 25 labeled examples recover 90% of full-data AUC on GPT-2. Steering shifts hallucination rates by 9.1 points on six models, erasure drops detection to chance, and distributed alignment search, the only method that bounds rank, localizes at most five causal dimensions. The internal advantage is regime-specific: probes far outperform P(True) and semantic entropy on adversarial misconceptions but tie on standard QA. Single-dataset probes transfer near-randomly until joint multi-dataset training restores 0.73-0.91 AUC. That centroid distance matches probe performance indicates class separation is a mean shift, making detection geometric rather than learned.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。