研究推理模型如何自我验证，发现关键注意力头和门控单元起核心作用。
The Geometry of Self-Verification in a Task-Specific Reasoning Model
- 通过结构化思维链分析模型自验证机制
- 仅3个注意力头即可关闭自验证功能
- 结果适用于基础模型与通用推理模型
推理模型如何验证自身答案？我们通过在CountDown任务上使用DeepSeek R1的训练方法，研究这一问题。利用偏好微调导致模式坍缩的特性，得到一个始终生成高度结构化思维链的模型。采用自上而下与自下而上的分析方法，逆向解析模型的输出验证机制。自上而下发现：门控线性单元（GLU）权重编码了如"success"或"incorrect"等验证相关标记；自下而上发现：前一词注意力头在本设置中主要负责自验证。两者结合：受层间通信通道启发，利用识别出的GLU权重定位到最少三个可关闭自验证的注意力头，指向可能更大的验证电路中的必要组件。最后验证，类似验证组件也存在于基础模型及通用推理模型DeepSeek-R1中。
原文摘要 · Abstract (English)
How do reasoning models verify their own answers? We study this question by training a model using DeepSeek R1's recipe on the CountDown task. We leverage the fact that preference tuning leads to mode collapse, yielding a model that always produces highly structured chain-of-thought sequences. With this setup, we do top-down and bottom-up analyses to reverse-engineer how the model verifies its outputs. Top-down, we find Gated Linear Unit (GLU) weights encoding verification-related tokens, such as ``success'' or ``incorrect''. Bottom-up, we find that ``previous-token heads'' are mainly responsible for self-verification in our setup. Our analyses meet in the middle: drawing inspiration from inter-layer communication channels, we use the identified GLU weights to localize as few as three attention heads that can disable self-verification, pointing to a necessary component of a potentially larger verification circuit. Finally, we verify that similar verification components exist in our base model and a general reasoning DeepSeek-R1 model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。