通过内部意图空间分析大模型推理过程，发现思维链提升准确率但引发不同模型的不确定性变化。
Intention Collapse: Intention-Level Metrics for Reasoning in Language Models
- 定义意图熵、有效维度和可恢复性三类无需训练的度量指标，评估模型内部状态
- 思维链使平均准确率从34.2%提升至47.3%，但在部分数据集上反而降低表现
- 同一思维链策略下，不同模型内部不确定性变化相反，且内部信号与最终输出不一致
语言生成将高维内部状态映射为单一词元序列。本文从意图坍缩视角研究这一多对一映射：从内部意图空间 I 到外部语言空间 L 的投影。提出三种低成本、模型无关的度量，基于坍缩前状态 I 计算：(i) 意图熵 Hint(I)，(ii) 有效维度 deff(I)，(iii) 可恢复性 Recov(I)，以探测器 AUROC 预测最终成功。在三个模型（Mistral-7B、LLaMA-3.1-8B、Qwen-2.5-7B）与三个基准（GSM8K、ARC-Challenge、AQUA-RAT）组成的3×3实验中，对比基线、思维链（CoT）与随机唠叨控制组（每组n=200）。CoT使平均准确率从34.2%提升至47.3%（+13.1个百分点），主要得益于GSM8K上的显著增益，但在ARC-Challenge上持续下降。各模型中，CoT诱导出不同的熵态：Mistral的dH = Hint(CoT) - Hint(Base) < 0（熵降低），而LLaMA的dH > 0（熵升高），揭示了思维链引发内部不确定性的异质性。探测器AUROC在部分设置中显著高于随机水平，并能与行为准确率解耦（例如，Qwen在ARC-Challenge上出现高可恢复性但低思维链准确率），表明内部信息丰富未必转化为最终离散决策。
原文摘要 · Abstract (English)
Language generation maps a rich, high-dimensional internal state to a single token sequence. We study this many-to-one mapping through the lens of intention collapse: the projection from an internal intention space I to an external language space L. We introduce three cheap, model-agnostic metrics computed on a pre-collapse state I: (i) intention entropy Hint(I), (ii) effective dimensionality deff(I), and (iii) recoverability Recov(I), operationalized as probe AUROC for predicting eventual success. We evaluate these metrics in a 3x3 study across models (Mistral-7B, LLaMA-3.1-8B, Qwen-2.5-7B) and benchmarks (GSM8K, ARC-Challenge, AQUA-RAT), comparing baseline, chain-of-thought (CoT), and a babble control (n=200 items per cell). CoT increases average accuracy from 34.2% to 47.3% (+13.1 pp), driven by large gains on GSM8K but consistent degradations on ARC-Challenge. Across models, CoT induces distinct entropy regimes relative to baseline, dH = Hint(CoT) - Hint(Base): Mistral shows dH < 0 (lower-entropy CoT), whereas LLaMA shows dH > 0 (higher-entropy CoT), highlighting heterogeneity in CoT-induced internal uncertainty. Finally, probe AUROC is significantly above chance in a subset of settings and can dissociate from behavioral accuracy (e.g., high AUROC alongside lower CoT accuracy on ARC-Challenge for Qwen), suggesting that informative internal signal is not always reliably converted into a final discrete decision under constrained response formats.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。