发现大模型推理潜力的关键微观指标:能否区分正确与错误推理
Soundness-Aware Level: A Microscopic Signature that Predicts LLM Reasoning Potential
- 用隐空间特征构建逻辑链,按语义严谨性分类规则
- 高潜力模型对严谨/混乱规则的内部分布差异显著(R²=0.87)
- 适合研究模型预训练机制或选型时评估推理能力
强化学习结合可验证奖励(RLVR)能激发大语言模型(LLM)的强推理能力,但不同基础模型在经过RLVR后的表现差异巨大。这引发根本问题:预训练模型的何种微观特性导致这种差异?我们形式化推理为由跨层稀疏自编码器(SAEs)提取的隐空间特征构成的命题链(“如果-那么”规则),估计特征间的转移概率,并利用大模型对每条规则的语义严谨性(如严格、合理、噪声)进行分类。关键发现:高潜力模型具备严谨性感知能力——其内部概率分布随规则严谨性系统性变化,对“严格”与“噪声”规则的分布分离明显;而弱模型则无视严谨性,所有规则对应同一分布。为此提出声学感知水平(SAL)指标,基于Jensen-Shannon散度量化分布分离程度。实验显示,SAL对多种模型族(Qwen、Mistral、Llama、DeepSeek)和规模(0.5B-14B)的后置RLVR推理性能预测符合精确经验定律(R²=0.87)。表明模型的推理潜力源于其预训练阶段对真伪知识的内在区分能力。研究揭示预训练在塑造推理中的核心作用,并提供一个基于内部机制的实用评估指标。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) can elicit strong reasoning in large language models (LLMs), while their performance after RLVR varies dramatically across different base models. This raises a fundamental question: what microscopic property of pre-trained models leads to this variation? To investigate, we formalize reasoning as chains of Horn clauses ("if-then" rules) built from features extracted from the LLM's latent space via cross-layer sparse autoencoders (SAEs). We estimate the transition probabilities between its features, and further categorize each rule by its semantic soundness level (e.g., strict, plausible, noisy) with an LLM. Our key discovery is that high-potential models are inherently soundness-aware: their internal probability distributions systematically shift across rules' soundness levels, becoming highly distinct for "strict" versus "noisy" rules. In contrast, weaker models are soundness-agnostic, collapsing to one distribution regardless of soundness levels. To quantify this, we introduce the Soundness-Aware Level (SAL), a microscopic metric using the Jensen-Shannon Divergence to measure the separation between these distributions. We show that SAL's predictions of post-RLVR reasoning performance follow a precise empirical law (R^2=0.87) across diverse model families (Qwen, Mistral, Llama, DeepSeek) and scales (0.5B-14B). This reveals that a model's reasoning potential is tied to its intrinsic, pre-trained ability to distinguish sound knowledge from unsound ones. These findings underscore the critical role of model pre-training in shaping reasoning and offer a practical metric grounded in the model's internal mechanisms for selecting/designing stronger base models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。