arXiv:2605.22389cs.CL2026-05被引 3

用熵值筛选高质量推理数据,训练更高效

Unified Data Selection for LLM Reasoning

论文配图:Unified Data Selection for LLM Reasoning
图 1 · 摘自论文原文
  • 仅计算高熵词的熵值之和,快速评估推理质量
  • 仅用20%高分数据训练,效果接近全量数据
  • 适用于多种训练方式,显著降低计算开销

为提升大模型在复杂长链式推理任务中的训练效率,现有方法或计算成本过高,或难以可靠区分高质量与低质量推理样本。为此,我们提出无需训练的高熵和(HES)指标,通过求每个样本中最高熵(如前0.5%)词的熵值之和来量化推理质量。在监督微调(SFT)、拒绝微调(RFT)和强化学习(RL)三种主流训练范式中验证,HES均表现稳定且显著降低计算开销。在SFT中,使用前20% HES排名数据即可达到全数据集性能,而低HES数据则导致性能下降;在RFT中,基于HES的方法显著优于基线;在RL中,采用HES筛选的成功轨迹使模型学习到更强推理模式,明显优于其他方法。结果表明,HES是一种鲁棒、无需训练的统一方法,可高效构建高级推理能力。

原文摘要 · Abstract (English)

Effectively training Large Language Models (LLMs) for complex, long-CoT reasoning is often bottlenecked by the need for massive high-quality reasoning data. Existing methods are either computationally expensive or fail to reliably distinguish high- from low-quality reasoning samples. To address this, we propose High-Entropy Sum (HES), a training-free metric that quantifies reasoning quality by summing only the entropy of the top (e.g., 0.5\%) highest-entropy tokens in each reasoning sample. We validate HES across three mainstream training paradigms: Supervised Fine-tuning (SFT), Rejection Fine-tuning (RFT), and Reinforcement Learning (RL), with extensive results demonstrating its consistent effectiveness and significantly reduced computational overhead. In SFT, training on the top 20\% HES-ranked data matches full-dataset performance, while using the lowest-HES data degrades it. In RFT, our HES-based training approach significantly outperforms baseline methods. In RL, HES-selected successful trajectories enable the model to learn strong reasoning patterns, significantly surpassing other compared methods. Our findings establish HES as a robust, training-free metric that enables a unified, effective, and efficient method for developing advanced reasoning in LLMs.

大模型训练推理优化数据筛选

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。