用模型自评和不确定性信号，从答对的题中继续提升LLM性能。
Learning from Saturated Data: Signals Beyond Correctness for LLM Training
- 用模型自评答案优劣和分词熵作为质量信号替代正确性
- 在简单算术任务上性能提升最高达18.6%，显著优于传统微调
- 复杂任务需谨慎使用，部分信号可能反而降低效果
大型语言模型（LLMs）能力的提升导致许多基准测试和训练数据集趋于饱和。为此，我们探究：即使问题已实现完美准确率，是否仍能利用其进一步提升下游性能。方法上，将二元正确性替换为两种更细粒度的质量信号：(1) 模型自身对解法相对质量的成对自评；(2) 基于分词级别的熵值，以分词不确定性作为解法质量的代理。我们将这些信号整合进多种训练算法，并在Qwen3-1.7B-Base上评估。在仅训练简单算术任务时，基于质量的信号使性能最高提升18.6%，显著优于标准微调（SFT）。但在GSM8K上，增益较弱且高度依赖信号类型；例如，自评结果与更强外部评判者一致性差，甚至导致性能低于基础模型。总体表明，质量信号可从饱和题目中提取有效信息以改进基础模型，但应用于复杂任务需精细校准与深入研究。
原文摘要 · Abstract (English)
The growing capabilities of large language models (LLMs) have led to the saturation of many benchmarks and training datasets used to improve them. Motivated by this, we investigate whether questions solved with perfect empirical accuracy can nevertheless be used to improve downstream performance. To do so, we replace binary correctness with two sources of more fine-grained quality signals: (1) pairwise LLM self-judgments, in which the model evaluates the relative quality of its own solutions, and (2) token-level entropy, where token-level uncertainty is used as a proxy for solution quality. We incorporate these signals into several training algorithms and evaluate them on Qwen3-1.7B-Base. When training exclusively on a simple arithmetic task, quality-based signals improve performance by up to $18.6\%$ over the base model, substantially outperforming SFT. On GSM8K, however, gains are more modest and depend strongly on the quality signal. For instance, self-judgments show poor agreement with a stronger external judge and can even degrade performance below the base model. Overall, our results suggest that quality-based training can extract useful signal from saturated questions for base models, but that applying such signals to more complex tasks requires careful calibration and further study.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。