改用概率目标函数，让大模型微调后更懂泛化。
Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuum
- 用多种概率目标替代传统负对数似然，适配不同能力模型
- 强模型用低概率词降权目标，弱模型仍用NLL更优
- 实验覆盖8个模型、27个任务，发现性能关键在模型能力连续体
监督微调（SFT）是大语言模型后训练的标准方法，但泛化能力常受限。我们发现其默认目标负对数似然（NLL）在后训练场景下可能不最优——因模型已有任务先验，且标注可能冗长嘈杂。本文系统研究多种概率目标，揭示其在不同条件下的表现差异。通过在8个模型主干、27个基准和7个领域上的全面实验与消融分析，我们识别出决定目标表现的关键维度：模型能力连续体。在模型较强时，倾向先验的目标（如 -p, -p¹⁰, 截断变体）持续优于NLL；在模型较弱时，NLL占优；中间区域无单一最佳目标。理论分析进一步阐明目标间的转换机制，为按模型能力适配目标提供原则性依据。代码已开源。
原文摘要 · Abstract (English)
Supervised fine-tuning (SFT) is the standard approach for post-training large language models (LLMs), yet it often shows limited generalization. We trace this limitation to its default training objective: negative log likelihood (NLL). While NLL is classically optimal when training from scratch, post-training operates in a different paradigm and could violate its optimality assumptions, where models already encode task-relevant priors and supervision can be long and noisy. In this work, we systematically study various probability-based objectives and characterize when and why different objectives succeed or fail under varying conditions. Through comprehensive experiments and extensive ablation studies across 8 model backbones, 27 benchmarks, and 7 domains, we uncover a critical dimension that governs objective behavior: the model-capability continuum. Near the model-strong end, prior-leaning objectives that downweight low-probability tokens (e.g., $-p$, $-p^{10}$, thresholded variants) consistently outperform NLL; toward the model-weak end, NLL dominates; in between, no single objective prevails. Our theoretical analysis further elucidates how objectives trade places across the continuum, providing a principled foundation for adapting objectives to model capability. The code is available at https://github.com/GaotangLi/Beyond-Log-Likelihood.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。