arXiv:2503.23913cs.CL2025-03被引 5

用不确定性自适应加权,提升大模型自训练时的推理能力。

Entropy-Based Adaptive Weighting for Self-Training

  • 根据模型预测不确定性动态调整数据权重,聚焦难例
  • 在MATH上提升1%,GSM8K上再增1-2%
  • 适合想优化自训练效果的研究者和工程师

大型语言模型在数学问题求解方面的能力日益受到关注,利用自生成推理路径作为提升模型性能的潜力路径备受青睐。这类路径捕捉了逐步的逻辑过程,仅需正确答案即可监督。自训练方法在推理任务中已被证明有效,且无需外部模型或人工标注。然而,如何优化自生成数据的使用仍是开放挑战。本文提出熵基自适应加权自训练方法(EAST),通过可调参数控制权重函数陡度,为模型不确定程度更高的样本分配更高权重,引导模型聚焦更具信息量和挑战性的例子,从而增强其推理能力。我们在GSM8K和MATH基准上评估该方法。实验结果表明,原始方法在MATH上几乎无提升(0%),而EAST相比基线模型获得约1%的增益;在GSM8K上,EAST相较原方法进一步提升了1-2%。

原文摘要 · Abstract (English)

The mathematical problem-solving capabilities of large language models have become a focal point of research, with growing interests in leveraging self-generated reasoning paths as a promising way to refine and enhance these models. These paths capture step-by-step logical processes while requiring only the correct answer for supervision. The self-training method has been shown to be effective in reasoning tasks while eliminating the need for external models and manual annotations. However, optimizing the use of self-generated data for model training remains an open challenge. In this work, we propose Entropy-Based Adaptive Weighting for Self-Training (EAST), an adaptive weighting strategy designed to prioritize uncertain data during self-training. Specifically, EAST employs a mapping function with a tunable parameter that controls the sharpness of the weighting, assigning higher weights to data where the model exhibits greater uncertainty. This approach guides the model to focus on more informative and challenging examples, thereby enhancing its reasoning ability. We evaluate our approach on GSM8K and MATH benchmarks. Empirical results show that, while the vanilla method yields virtually no improvement (0%) on MATH, EAST achieves around a 1% gain over backbone model. On GSM8K, EAST attains a further 1-2% performance boost compared to the vanilla method.

自训练推理能力加权策略

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。