超拟合能提升大模型生成质量，因末层特征空间扩展而非温度调节。
Beyond Temperature: Hyperfitting as a Late-Stage Geometric Expansion
- 末层特征空间扩展约80.8维，动态重排词序增强长尾词生成。
- 温度调节无法复现超拟合的多样性提升，说明机制不同。
- 仅微调最后5层的晚期LoRA策略，参数少且效果强。
近期研究发现一种反直觉现象‘超拟合’：将大语言模型在小数据集上微调至接近零训练损失，意外提升了开放式生成质量并缓解贪婪解码中的重复问题。尽管有效，其内在机制仍不清晰，极低熵输出分布暗示可能等同于简单的温度调节。本文表明该现象本质不同于分布锐化；熵匹配对照实验显示温度调节无法复制超拟合带来的多样性增益。此外，消融实验证伪了静态词汇重加权假设，揭示超拟合依赖动态、上下文相关的词序重排机制。层间分析定位该效应发生在最后一层Transformer块的‘终端扩展’中，特征空间显著扩展（Delta Dim ≈ +80.8），促进深层尾部词的生成。我们进一步提出晚期LoRA，仅更新最后5层，实现鲁棒生成且参数更新极少。
原文摘要 · Abstract (English)
Recent work has identified a counterintuitive phenomenon termed "Hyperfitting", where fine-tuning Large Language Models (LLMs) to near-zero training loss on small datasets surprisingly enhances open-ended generation quality and mitigates repetition in greedy decoding. While effective, the underlying mechanism remains poorly understood, with the extremely low-entropy output distributions suggesting a potential equivalence to simple temperature scaling. In this work, we demonstrate that this phenomenon is fundamentally distinct from distribution sharpening; entropy-matched control experiments reveal that temperature scaling fails to replicate the diversity gains of hyperfitting. Furthermore, we falsify the hypothesis of static vocabulary reweighting, showing through ablation studies that hyperfitting relies on a dynamic, context-dependent rank reordering mechanism. Layer-wise analysis localizes this effect to a "Terminal Expansion" in the final transformer block, where a substantial geometric expansion of the feature space (Delta Dim approx +80.8) facilitates the promotion of deep-tail tokens. Additionally, we introduce Late-Stage LoRA, a targeted fine-tuning strategy that updates only the final 5 layers, yielding robust generation with minimal parameter updates
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。