让大模型自己判断该学什么,提升强化微调效率与效果
Internalizing Curriculum Judgment for LLM Reinforcement Fine-Tuning

- 用模型自身训练结果预测提示信息量,实现内部化课程判断
- 在数学推理、代码生成等任务中提速67%,收敛更快
- 无需人工设计规则或额外模型,适合追求高效微调的研究者
在大模型强化微调(RFT)中,课程学习能显著提升效率与性能。现有方法依赖人工设计规则或辅助模型进行课程判断,易与策略训练动态脱节。本文提出METIS(元认知内生自判断)框架,将课程判断能力内化为模型原生属性。基于一个关键观察:提示内的奖励方差可有效衡量提示的信息量,METIS通过近期训练结果生成轻量级上下文学习示例来预测该指标,并据此动态分配训练资源。同时,METIS通过联合优化标准RFT奖励与自判断奖励,构建判断与优化的闭环,使模型具备‘知道该学什么’的元认知能力。在涵盖数学推理、代码生成、代理函数调用等离散与连续任务的广泛基准上,METIS持续表现更优,且收敛速度最快提升67%。该方法摆脱了人工规则和额外模型,建立了一个简单、闭环、高效的内生课程学习范式。
原文摘要 · Abstract (English)
In LLM Reinforcement Fine-Tuning (RFT), curriculum learning drives both efficiency and performance. Yet, current methods externalize curriculum judgment via handcrafted heuristics or auxiliary models, risking misalignment with the policy's training dynamics. In this paper, we introduce METIS (METacognitive Internalized Self-judgment), a novel framework that internalizes curriculum judgment as a native capability. Leveraging a critical observation that within-prompt reward variance effectively gauges prompt informativeness, METIS predicts this metric based on recent training outcomes as lightweight in-context learning examples. This intrinsic self-judgment then dynamically dictates the training allocation. Moreover, METIS closes the loop between judgment and optimization by jointly optimizing the standard RFT rewards and a self-judgment reward. This allows the policy to learn what to learn next, as a form of metacognition. Across extensive discrete and continuous RFT benchmarks from mathematical reasoning, code generation, to agentic function-calling, METIS consistently delivers superior performance while accelerating convergence by up to 67%. By bypassing handcrafted heuristics and auxiliary models, our work establishes a simple, closed-loop, and highly efficient curriculum internalization paradigm for LLM reinforcement fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。