自适应采样让模型更高效地从错误中学习。
AdaSTaR: Adaptive Data Sampling for Training Self-Taught Reasoners
- 根据难度和多样性动态调整训练数据采样。
- 6个基准测试全胜，平均减少58.6%计算量。
- 适合想提升推理模型训练效率的研究者。
自教式推理器（STaR），又称拒绝采样微调（RFT），是自进化推理语言模型训练流程中的关键环节。现有方法常采用随机数据采样，导致训练不平衡：对已解决例题过度训练，而对困难样本关注不足。为此，我们提出自适应STaR（AdaSTaR），引入两种自适应采样机制：（1）多样性采样，确保各类型样本均衡训练；（2）课程式采样，动态匹配模型当前能力调整数据难度。在六个基准测试中，AdaSTaR实现全部6项最佳测试准确率，相比多种基线平均降低58.6%的训练浮点运算量。该方法在不同预训练语言模型及更大规模模型上均表现优异，为更高效、更有效的自进化语言模型提供了新路径。
原文摘要 · Abstract (English)
Self-Taught Reasoners (STaR), synonymously known as Rejection sampling Fine-Tuning (RFT), is an integral part of the training pipeline of self-improving reasoning Language Models (LMs). The self-improving mechanism often employs random observation (data) sampling. However, this results in trained observation imbalance; inefficiently over-training on solved examples while under-training on challenging ones. In response, we introduce Adaptive STaR (AdaSTaR), a novel algorithm that rectifies this by integrating two adaptive sampling principles: (1) Adaptive Sampling for Diversity: promoting balanced training across observations, and (2) Adaptive Sampling for Curriculum: dynamically adjusting data difficulty to match the model's evolving strength. Across six benchmarks, AdaSTaR achieves best test accuracy in all instances (6/6) and reduces training FLOPs by an average of 58.6% against an extensive list of baselines. These improvements in performance and efficiency generalize to different pre-trained LMs and larger models, paving the way for more efficient and effective self-improving LMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。