用信息瓶颈缓解代码微调中的记忆陷阱，提升模型泛化能力
Breaking Memorization Barriers in LLM Code Fine-Tuning via Information Bottleneck for Improved Generalization
- 在微调时加入信息瓶颈约束，压缩冗余记忆特征
- 在OriGen和Evol-CodeAlpaca-V1上提升Pass@$1$性能，稳定提升Pass@$k^{(m)}$
- 适合追求高泛化性、抗过拟合的代码生成研究者
通过监督微调（FT）将预训练大语言模型适配到代码领域已广泛应用。但我们发现一种此前被低估的失效模式——记忆障碍：基础模型对下游代码数据的强记忆会阻碍优化，使标准微调难以获取新的通用代码知识。为此，我们提出信息瓶颈（IB）引导的微调方法（IB-FT），对代码数据的隐藏表示施加IB惩罚，压缩冗余记忆特征，同时保留任务相关信息。在两个代码基准（OriGen和Evol-CodeAlpaca-V1）上的大量实验表明，IB-FT显著缓解了记忆障碍，提升了Top-1性能（Pass@$1$），并在更严格的多样本指标Pass@$k^{(m)}$（要求k个采样中至少m个通过单元测试）下实现更稳定的增益。
原文摘要 · Abstract (English)
Adapting pretrained large language models (LLMs) to code domains via supervised fine-tuning (FT) has been commonly used for code generation. However, we identify a previously underappreciated failure mode, the memorization barrier, where strong memorization of downstream code data in the base model could trap optimization and prevent the standard FT from effectively acquiring new, generalizable code knowledge. To overcome this barrier, we propose the information bottleneck (IB)-guided fine-tuning, termed IB-FT, which applies an IB penalty on hidden representations of the code data to compress spurious, memorized features while preserving task-relevant information. Extensive experiments on two code benchmarks (OriGen and Evol-CodeAlpaca-V1) show that IB-FT substantially alleviates the memorization barrier, improves top-1 performance (Pass@$1$), and yields far more stable gains under the stricter multi-sample metric Pass@$k^{(m)}$ (a problem counts as solved only if at least $m$ of $k$ samples pass unit tests) compared with conventional FT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。