发现Transformer在推理中突然锁定答案的稳定层，提升准确率并揭示模型决策机制。
The Hard Decision Layer: Evidence for Committed Inference in Transformers
- 识别出答案排名骤然稳定的'硬决策层'，无需额外训练路由策略。
- 在四模型四数据集上验证，该层出现后准确率最高提升0.61且性能趋于稳定。
- 现象与微调无关，适用于不同标签格式和题型复杂度，适合研究模型推理机制者。
我们研究基于Transformer的语言模型在多项选择题问答中何时及如何做出预测。通过实证分析，在Qwen、Llama、Granite、Mistral四种语言模型和四个基准数据集上，均发现答案选项排名在推理过程中突然稳定的现象，这一现象被称为'硬决策层'（HDL）。该现象无需学习路由策略即可自然涌现。实验还表明，该层对微调具有不变性。在硬决策层处，模型准确率显著提升，最高达+0.61（Qwen在CommonsenseQA上），之后性能趋于稳定。系统性消融实验验证了该现象对标签格式和问题复杂度的鲁棒性，证明其为模型架构的本质特征。研究结果为理解Transformer推理机制提供了机理洞察，并为高效推理与模型调控提供了新思路。所有代码与复现所需结果见https://github.com/Mystic-Slice/hard-decision-layer。
原文摘要 · Abstract (English)
We investigate where and how transformer-based language models commit to predictions in multiple-choice question answering. We identify the _Hard Decision Layer_ (HDL), a natural architectural property where answer option rankings stabilize abruptly during inference. Empirical validation across four language models (Qwen, Llama, Granite, Mistral) and four benchmark datasets demonstrates consistent HDL emergence without learned routing policies. We also show that the HDL is invariant to fine-tuning. Our results reveal striking accuracy improvements at the HDL: up to +0.61 (Qwen on CommonsenseQA), after which performance stabilizes. Systematic ablations on label formats and problem complexity confirm the phenomenon is fundamental to model architecture. These findings offer mechanistic insights into transformer inference and suggest opportunities for efficient reasoning and model steering. All code and results required to reproduce this work are available in https://github.com/Mystic-Slice/hard-decision-layer
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。