通过筛选高价值词元提升语言模型对齐效果
ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selection
- 基于词元概率选择核心语义内容，屏蔽可替换表达
- 在数学与推理任务上超越传统微调方法
- 适合追求高效精准对齐的模型优化场景
监督微调（SFT）是将大语言模型对齐人类意图的基础后训练策略。然而，传统SFT常忽略语言的一对多特性，强制模型仅对齐单一参考答案，导致模型过度拟合非核心表达。尽管实证分析表明引入多个参考答案可缓解此问题，但高昂的数据与计算成本要求我们优先解决单参考过拟合问题。基于此，我们揭示了词元概率与语义重要性的内在关联：高概率词元承载核心逻辑框架，低概率词元多为可替换表达。据此提出ProFit，通过有选择地掩码低概率词元，防止表面过拟合。大量实验表明，ProFit在通用推理与数学基准测试中持续优于传统SFT基线。
原文摘要 · Abstract (English)
Supervised fine-tuning (SFT) is a fundamental post-training strategy to align Large Language Models (LLMs) with human intent. However, traditional SFT often ignores the one-to-many nature of language by forcing alignment with a single reference answer, leading to the model overfitting to non-core expressions. Although our empirical analysis suggests that introducing multiple reference answers can mitigate this issue, the prohibitive data and computational costs necessitate a strategic shift: prioritizing the mitigation of single-reference overfitting over the costly pursuit of answer diversity. To achieve this, we reveal the intrinsic connection between token probability and semantic importance: high-probability tokens carry the core logical framework, while low-probability tokens are mostly replaceable expressions. Based on this insight, we propose ProFit, which selectively masks low-probability tokens to prevent surface-level overfitting. Extensive experiments confirm that ProFit consistently outperforms traditional SFT baselines on general reasoning and mathematical benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。