通过筛选正确回答提升大模型泛化能力，避免微调后过度专精。
Selective Self-to-Supervised Fine-Tuning for Generalization in Large Language Models
- 基于模型自身正确回答筛选样本，减少微调时的过拟合。
- 在多个基准上将性能下降从4.4降至2.5，泛化能力更强。
- 适合追求模型稳定性和多任务适应性的研究者使用。
在特定数据集上微调大语言模型是提升目标任务表现的常用方法，但常导致过拟合，使模型过度依赖任务或训练数据特征，丧失泛化能力。本文提出选择性自监督微调（S3FT），在保持优于标准监督微调（SFT）性能的同时，提升了模型泛化能力。S3FT利用同一问题存在多种有效回答的特性，通过部署合适判别器识别训练集中模型的正确回答，再以这些正确回答和标准答案（或其改写）对剩余样本进行微调。实验在数学推理、Python编程和阅读理解任务上验证了该方法的有效性。结果显示，标准SFT在MMLU和TruthfulQA等基准上平均性能下降达4.4；而S3FT将该下降减半至2.5，表明其在提升任务性能的同时显著增强泛化能力。
原文摘要 · Abstract (English)
Fine-tuning Large Language Models (LLMs) on specific datasets is a common practice to improve performance on target tasks. However, this performance gain often leads to overfitting, where the model becomes too specialized in either the task or the characteristics of the training data, resulting in a loss of generalization. This paper introduces Selective Self-to-Supervised Fine-Tuning (S3FT), a fine-tuning approach that achieves better performance than the standard supervised fine-tuning (SFT) while improving generalization. S3FT leverages the existence of multiple valid responses to a query. By utilizing the model's correct responses, S3FT reduces model specialization during the fine-tuning stage. S3FT first identifies the correct model responses from the training set by deploying an appropriate judge. Then, it fine-tunes the model using the correct model responses and the gold response (or its paraphrase) for the remaining samples. The effectiveness of S3FT is demonstrated through experiments on mathematical reasoning, Python programming and reading comprehension tasks. The results show that standard SFT can lead to an average performance drop of up to $4.4$ on multiple benchmarks, such as MMLU and TruthfulQA. In contrast, S3FT reduces this drop by half, i.e. $2.5$, indicating better generalization capabilities than SFT while performing significantly better on the fine-tuning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。