通过自适应熵正则化提升大模型微调多样性，改善强化学习效果。
SED-SFT: Selectively Encouraging Diversity in Supervised Fine-Tuning
- 引入选择性掩码机制，在优化目标中动态增强生成多样性。
- 在8个数学基准上显著提升多样性，对齐后强化学习性能提升2.06和1.20点。
- 计算开销极低，适合部署在资源受限的微调场景，尤其适合数学推理类任务。
监督微调（SFT）结合强化学习（RL）已成为大语言模型（LLM）后训练的标准范式。然而，传统基于交叉熵（CE）损失的SFT常导致模式坍缩，使模型过度集中于特定回复模式，严重限制后续强化学习的探索效率。尽管近期研究尝试用其他损失函数替代CE以维持多样性或优化更新策略，但难以在多样性和准确性之间取得平衡，导致强化学习阶段表现不佳。为此，我们提出SED-SFT，该框架基于词元探索空间自适应地鼓励多样性。其核心是在优化目标中引入选择性熵正则化项与选择性掩码机制。在八个数学基准上的大量实验表明，相较于传统CE损失，SED-SFT在几乎无额外计算开销下显著提升生成多样性，在Llama-3.2-3B-Instruct和Qwen2.5-Math-7B-Instruct上分别带来2.06和1.20分的强化学习性能提升。代码已公开于https://github.com/pppa2019/SED-SFT。
原文摘要 · Abstract (English)
Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) has emerged as the standard post-training paradigm for large language models (LLMs). However, the conventional SFT process, driven by Cross-Entropy (CE) loss, often induces mode collapse, where models over-concentrate on specific response patterns. This lack of distributional diversity severely restricts the exploration efficiency required for subsequent RL. While recent studies have attempted to improve SFT by replacing the CE loss, aiming to preserve diversity or refine the update policy, they fail to adequately balance diversity and accuracy, thereby yielding suboptimal performance after RL. To address the mode collapse problem, we propose SED-SFT, which adaptively encourages diversity based on the token exploration space. This framework introduces a selective entropy regularization term with a selective masking mechanism into the optimization objective. Extensive experiments across eight mathematical benchmarks demonstrate that SED-SFT significantly enhances generation diversity with a negligible computational overhead increase compared with CE loss, yielding average improvements of 2.06 and 1.20 points in subsequent RL performance over standard CE-based baselines on Llama-3.2-3B-Instruct and Qwen2.5-Math-7B-Instruct, respectively. The code is publicly available at https://github.com/pppa2019/SED-SFT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。