模型生成代码前的隐藏状态已能预判代码正确性。
Code Correctness Is Linearly Decodable from LLM Hidden States Before Generation
- 用线性探测法在生成前的隐藏状态中预测代码正确性。
- 准确率AUC达0.881，且不受提示长度干扰。
- 为评估大模型生成能力提供新诊断方法，适合模型分析者。
大型语言模型在隐藏状态中编码丰富信息。本文研究Qwen3-4B-Instruct-2507模型在未生成代码前，其隐藏状态是否已包含代码正确性的可解码信号，基于LiveCodeBench中的444个任务进行评估。结果显示，首次尝试代码的正确性可在线性探测下从最终提示词对应的隐藏状态中解码，泄漏无关的50次外部分割中平均AUC为0.881 ± 0.008。为排除提示长度影响，对每个隐藏维度进行线性残差化处理，探测仍达AUC 0.842 ± 0.010，显著高于仅依赖提示长度的逻辑回归基线（0.657 ± 0.014），且所有非线性模型均未提升表现。关于自修复是否留下几何痕迹的问题因成功修复样本过少无法验证。该研究兼具实证与方法论价值，证明生成前隐藏状态蕴含稳健的代码正确性信号，并提供控制混淆因素的诊断工具。
原文摘要 · Abstract (English)
Large language models encode rich information in their hidden states. This work asks whether the correctness of code that Qwen3-4B-Instruct-2507 has not yet generated is already legible in its hidden states, evaluated on a set of 444 tasks from LiveCodeBench. The correctness of the model's first-attempt code is linearly decodable from the hidden state at the final prompt token, captured before any output token is generated, with a leakage-free held-out AUC of 0.881 +/- 0.008 across 50 outer splits. To assess whether this signal is explained by prompt length, each hidden state dimension is residualized with respect to its linear effect. The probe still achieves an AUC of 0.842 +/- 0.010, substantially above a logistic prompt-length baseline of 0.657 +/- 0.014, and none of the nonlinear models tested improves upon it. A companion question about whether self-repair leaves a geometric signature in the model's hidden states could not be answered, because successful repairs following a failed first attempt are too rare in this setting to support the analysis. The contribution is both empirical and methodological, providing evidence that pre-generation hidden states contain a robust signal of eventual code correctness, together with a confound-control diagnostic that quantifies how much of that signal survives adjustment for prompt length.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。