通过选择性自重演，提升大模型在微调后的泛化能力。
Selective Self-Rehearsal: A Fine-Tuning Approach to Improve Generalization in Large Language Models
- 利用模型自身正确回答作为监督信号，减少过拟合
- 在多个基准上仅平均下降2%，相比标准微调降低14.7个百分点
- 适合需要保持泛化能力的下游任务场景
在特定数据集上微调大语言模型是提升目标任务性能的常见方法，但常导致过拟合，使模型过度适应任务或训练数据特征，从而损失泛化能力。本文提出选择性自重演（Selective Self-Rehearsal, SSR），在性能接近标准监督微调（SFT）的同时提升泛化性。SSR利用同一问题存在多个有效回答的事实，通过部署一个合适的LLM作为裁判，从训练集中识别出模型的正确回答，然后使用这些正确回答与真实答案共同对模型进行微调。在判断不可回答问题的任务上进行实验，结果表明，标准SFT在多个基准（如MMLU和TruthfulQA）上平均性能下降高达16.7%；而SSR平均下降仅约2%，显著改善了泛化能力。
原文摘要 · Abstract (English)
Fine-tuning Large Language Models (LLMs) on specific datasets is a common practice to improve performance on target tasks. However, this performance gain often leads to overfitting, where the model becomes too specialized in either the task or the characteristics of the training data, resulting in a loss of generalization. This paper introduces Selective Self-Rehearsal (SSR), a fine-tuning approach that achieves performance comparable to the standard supervised fine-tuning (SFT) while improving generalization. SSR leverages the fact that there can be multiple valid responses to a query. By utilizing the model's correct responses, SSR reduces model specialization during the fine-tuning stage. SSR first identifies the correct model responses from the training set by deploying an appropriate LLM as a judge. Then, it fine-tunes the model using the correct model responses and the gold response for the remaining samples. The effectiveness of SSR is demonstrated through experiments on the task of identifying unanswerable queries across various datasets. The results show that standard SFT can lead to an average performance drop of up to $16.7\%$ on multiple benchmarks, such as MMLU and TruthfulQA. In contrast, SSR results in close to $2\%$ drop on average, indicating better generalization capabilities compared to standard SFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。