模型自改进时易陷入表面提升，新方法通过融合权重保持泛化能力。
Superficial Self-Improved Reasoners Benefit from Model Merging
- 通过迭代融合原始与自改进模型权重，保留真实推理能力。
- 自改进后模型在域内准确率提升，但跨域泛化能力反而下降。
- 适合关注大模型自进化稳定性的研究者和实践者。
随着语言模型规模扩大，其推理能力接近人类水平，自改进成为生成高质量数据的有效途径。然而，以往研究指出自改进存在模型坍塌风险——输出趋于确定性。我们发现更根本的问题是‘表面自改进’现象：尽管模型在域内（ID）任务上推理准确率提升，却因记忆而非真正理解，导致域外（OOD）任务泛化能力下降。系统性分析显示，自改进过程中模型权重更新集中在非关键推理层，造成浅层学习。为此，我们提出迭代模型融合（IMM），通过策略性结合原始模型与自改进模型的权重，在保留泛化能力的同时融入真实推理提升。该方法有效缓解了模型坍塌与表面学习问题，推动更稳定的自改进系统发展。
原文摘要 · Abstract (English)
As scaled language models (LMs) approach human-level reasoning capabilities, self-improvement emerges as a solution to synthesizing high-quality data corpus. While previous research has identified model collapse as a risk in self-improvement, where model outputs become increasingly deterministic, we discover a more fundamental challenge: the superficial self-improved reasoners phenomenon. In particular, our analysis reveals that even when LMs show improved in-domain (ID) reasoning accuracy, they actually compromise their generalized reasoning capabilities on out-of-domain (OOD) tasks due to memorization rather than genuine. Through a systematic investigation of LM architecture, we discover that during self-improvement, LM weight updates are concentrated in less reasoning-critical layers, leading to superficial learning. To address this, we propose Iterative Model Merging (IMM), a method that strategically combines weights from original and self-improved models to preserve generalization while incorporating genuine reasoning improvements. Our approach effectively mitigates both LM collapse and superficial learning, moving towards more stable self-improving systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。