模型推理失败常因输出瓶颈，内部逻辑仍完好。
Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores
- 用极简无标签修正方法诊断模型内部逻辑
- 仅25个样本即可恢复9-34分准确率提升
- 适合研究模型隐式推理能力的学者
当大语言模型在推理任务中失败时，通常被认为缺乏相应能力。然而，这可能混淆了真实能力缺失与晚期输出瓶颈。我们在多个推理基准上观察到一致的读出差距：即使原生序列评分因结构偏差完全崩溃，隐藏状态探测仍能解码正确答案。为检验实例特定逻辑是否在崩溃中存活，我们引入一种基于最小化、无目标标签的加性修正诊断协议。仅需在25个无标签样本上拟合两个参数，即可使Qwen3.5模型准确率恢复9–34个百分点，并成功迁移到OLMo-2-1B和Llama-3.1-8B。关键的是，这些恢复的决策在简单词法重叠无法解决的难题上依然有效，且显著超越保持计数的置换基线。结果表明，许多看似零样本推理缺陷实为表达失败，掩盖了完整的内部逻辑，呼吁对基准评估采用更精准的解释。
原文摘要 · Abstract (English)
When a large language model fails a reasoning task, it is often assumed to lack the underlying capability. However, this conflates a genuine absence of reasoning with a late-stage output bottleneck. We observe a consistent readout gap across diverse reasoning benchmarks: hidden-state probes successfully decode correct answers even when native sequence scoring completely collapses due to structural biases. To test whether instance-specific logic survives this collapse, we introduce a diagnostic protocol using a minimal, target-label-free additive correction. Fitting just two parameters on as few as 25 unlabeled examples recovers 9--34 accuracy points for Qwen3.5 models, transferring successfully to OLMo-2-1B and Llama-3.1-8B. Crucially, these recovered decisions persist on hard instances unresolved by simple lexical overlap and significantly exceed count-preserving permutation baselines. Our results show that many apparent zero-shot reasoning deficits are expression failures masking intact internal logic, urging a narrower interpretation of benchmark evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。