无需人工标注,自动发现大模型推理中的隐藏行为模式。
Fantastic Reasoning Behaviors and Where to Find Them: Unsupervised Discovery of the Reasoning Process
- 用稀疏自编码器从推理步骤中挖掘可解释的推理向量。
- 能分离出反思、回溯等行为,并控制其强度而不需重训练。
- 发现人类未定义的新行为,如影响回答置信度的向量。
尽管近期大语言模型具备强大推理能力,但其内部推理机制仍不清晰。以往方法依赖人工定义的概念(如过度思考、反思)进行有监督分析,难以覆盖所有潜在推理行为,尤其在标记空间中难以定义。本文提出无监督框架RISE(通过稀疏自编码器实现推理行为可解释性),通过将思维链分解为句级“步骤”,在步骤级激活上训练稀疏自编码器(SAEs),发现编码不同推理行为的方向向量。可视化与聚类分析显示这些行为在解码器列空间中占据分离区域。对SAE生成向量进行定向干预,可控制性增强或抑制特定行为,改变推理路径而无需重训练。除了行为解耦,SAEs还捕捉到响应长度等结构特征,揭示长/短推理轨迹聚类。更关键的是,该方法能发现人类未标注的新行为,例如识别出影响回答置信度的向量。这些结果表明,无监督潜在发现对理解与可控引导大模型推理具有重要潜力。
原文摘要 · Abstract (English)
Despite the growing reasoning capabilities of recent large language models (LLMs), their internal mechanisms during the reasoning process remain underexplored. Prior approaches often rely on human-defined concepts (e.g., overthinking, reflection) at the word level to analyze reasoning in a supervised manner. However, such methods are limited, as it is infeasible to capture the full spectrum of potential reasoning behaviors, many of which are difficult to define in token space. In this work, we propose an unsupervised framework (namely, RISE: Reasoning behavior Interpretability via Sparse auto-Encoder) for discovering reasoning vectors, which we define as directions in the activation space that encode distinct reasoning behaviors. By segmenting chain-of-thought traces into sentence-level 'steps' and training sparse auto-encoders (SAEs) on step-level activations, we uncover disentangled features corresponding to interpretable behaviors such as reflection and backtracking. Visualization and clustering analyses show that these behaviors occupy separable regions in the decoder column space. Moreover, targeted interventions on SAE-derived vectors can controllably amplify or suppress specific reasoning behaviors, altering inference trajectories without retraining. Beyond behavior-specific disentanglement, SAEs capture structural properties such as response length, revealing clusters of long versus short reasoning traces. More interestingly, SAEs enable the discovery of novel behaviors beyond human supervision. We demonstrate the ability to control response confidence by identifying confidence-related vectors in the SAE decoder space. These findings underscore the potential of unsupervised latent discovery for both interpreting and controllably steering reasoning in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。