将小模型注意力转为线性注意力后，模型只认选项标签不看内容，修复后准确率提升12.48点。
Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model
- 用线性注意力替换大部分注意力层，在消费级显卡上完成训练。
- 修复前模型仅凭选项标签选答案，正确率仅25-29%（原为50.6%）。
- 通过格式化指令微调修复标签依赖问题，适合关注模型可解释性的研究者。
我们在单张消费级显卡预算下，将Qwen3-0.6B-Base的28个全注意力层中的21个转换为KDA（Kimi Delta Attention）线性注意力层，并提出一个关键问题：转换究竟破坏了什么？手术后，隐藏状态对齐与端到端KL蒸馏使学生模型在困惑度上接近教师模型，但多项选择准确率仍维持在随机水平（C-Eval上25-29%，教师为50.6%）。通过四重排列诊断法（旋转选项但保持内容不变），我们发现模型固守选项标签（81%时间选“A”；161道题中有106道在所有旋转中保持相同标签），而非理解答案内容——这是标准蒸馏指标无法察觉的接口损伤。经过1,000步仅完成式格式目标的KL蒸馏阶段修复后，准确率提升12.48点，标签依赖现象大致减半；随后进行人格化SFT和一轮策略内DPO，基准得分保持在噪声范围内。我们开源代码、权重、训练配方及完整审计轨迹，并总结工程经验，包括在FP32主控模式下，bf16优化器更新被无声吞没的失败模式，这些是实现该预算下收敛的关键。
原文摘要 · Abstract (English)
We convert 21 of 28 full-attention layers of Qwen3-0.6B-Base into KDA (Kimi Delta Attention) linear-attention layers on a single consumer-grade GPU budget, and ask a simple question: what exactly does the conversion break? After surgery, hidden-state alignment and end-to-end KL distillation drive the student close to its teacher in perplexity, yet multiple-choice accuracy stays near random chance (25-29% vs. the teacher's 50.6% on C-Eval). Using a four-permutation diagnostic that rotates answer options while holding content fixed, we show the model sticks to option labels (predicting "A" 81% of the time; 106/161 questions keep the same label under all four rotations) rather than following answer content -- an interface injury that standard distillation metrics cannot see. A 1,000-step format-targeted completion-only KL stage repairs the interface (+12.48 points on C-Eval, label-stickiness roughly halved), after which persona SFT and one round of on-policy DPO preserve benchmark scores within noise. We release code, weights, recipes, and the full audit trail, and distill the engineering lessons -- including an FP32-master failure mode in which bf16 optimizer updates are silently swallowed -- that made convergence possible at this budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。