顶流模型在竞赛中严重过拟合，数据清洗比模型优化更重要
NeurIPS 2023 LLM Efficiency Fine-tuning Competition
- 用公开+封闭双阶段评估模型泛化能力
- 冠军方案靠数据清洗而非复杂算法
- 揭示评测基准过拟合问题，适合研究者复现
对 NeurIPS 2023 大语言模型微调竞赛的分析显示，表现最佳的模型在基准数据集上出现显著过拟合，反映出主流排行榜普遍存在基准过拟合问题。该竞赛分为公开任务阶段与未见任务的闭门评估阶段，用于检验微调后 LLM 的泛化能力。结果表明，当前基于基准的生成模型评估方法存在局限性，亟需更稳健的评测手段。值得注意的是，获胜方案均采用标准开源库，核心策略聚焦于数据清洗。为促进后续研究与可复现性，我们公开所有参赛代码、Docker 文件及评估框架，为社区研究大模型微调、过拟合与可复现性提供宝贵资源。
原文摘要 · Abstract (English)
Our analysis of the NeurIPS 2023 large language model (LLM) fine-tuning competition revealed the following trend: top-performing models exhibit significant overfitting on benchmark datasets, mirroring the broader issue of benchmark overfitting on popular leaderboards and that data curation is essential in order to get a high performing LLM. The competition, which consisted of two stages - an open evaluation stage with publicly available tasks and a closed evaluation stage with unseen tasks - allowed us to assess the generalizability of fine-tuned LLMs. Our results highlight the limitations of current benchmark-based evaluation schemes for generative models and demonstrate the need for more robust evaluation methods. Notably, the winning submissions utilized standard open-source libraries and focused primarily on data curation. To facilitate further research and promote reproducibility, we release all competition entries, Docker files, and evaluation infrastructure, providing a valuable resource for the community to explore fine-tuning, overfitting, and reproducibility in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。