用自我反思反馈提升大模型训练效率，早期即引入质量评估
Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stages
- 用语言反馈模型为数据打标签，指导训练优先级
- 在75亿至120亿参数模型上实现最高2.8倍算力效率提升
- 适合追求高效训练与高阶推理能力的开发者
我们研究如何更高效地扩展当前大模型训练流水线中多个不断增长的阶段。核心思路是：利用后期阶段（如后训练）的动态信息来优化前期阶段（如预训练）。为此，我们提出一种名为自省训练（Introspective X Training, IXT）的方法，受离线奖励条件强化学习启发，适用于任意训练阶段。IXT使用一个思维奖励模型，基于自然语言对数据生成批判性反馈，并以此标注数据，实现从训练初期就开始的质量感知训练。模型通过将生成的反馈作为前缀条件输入数据进行训练，确保并非所有标记都同等重要，显著早于传统方法引入质量区分机制。在75亿至120亿参数的Transformer密集型大模型上，从零开始训练至总共见过18万亿个令牌的全面实验表明，该方法使缩放曲线发生偏移，普遍实现高达2.8倍的计算效率提升；并在数学和代码等任务中达到以往方法无法企及的性能水平。
原文摘要 · Abstract (English)
We tackle the question of how to scale more efficiently across the many, ever-growing stages of current LLM training pipelines. Our guiding intuition stems from the fact that the dynamics of later stages of the pipeline, e.g. post-training, can be used to inform earlier stages such as pre-training. To this end, we propose Introspective Training (or IXT), inspired by offline reward-conditioned reinforcement learning and applicable to any stage of training. IXT uses a thinking reward model to annotate data with natural language critique based feedback, enabling quality aware training from the earliest stages of the pipeline. Models are then trained by prefix-conditioning the data with the generated feedback -- ensuring that not all tokens are treated equally starting much earlier in training than usual. Comprehensive experiments on 7.5-12B transformer-based dense LLMs trained from scratch all the way up to 18 Trillion tokens seen show that our method: bends scaling curves resulting in up to 2.8x more compute efficiency generally; and reaches performance levels unachievable for models trained otherwise in domains such as math and code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。