用元学习动态平衡模仿与探索,提升大模型推理能力。
AMFT: Aligning LLM Reasoners by Meta-Learning the Optimal Imitation-Exploration Balance
- 通过元梯度控制器自动调节模仿与探索的权重
- 在数学、视觉推理等任务上达到新SOTA
- 适合研究大模型对齐与高效训练的学者
大型语言模型通常通过监督微调(SFT)和强化学习(RL)两阶段流程进行推理任务微调,但存在灾难性遗忘及模仿与探索权衡不佳的问题。现有单阶段方法虽尝试统一SFT与RL,但缺乏动态平衡机制。本文从隐式奖励的理论视角出发,将SFT与RL视为互补的奖励信号,提出自适应元微调(AMFT),一种单阶段算法,通过可学习的元梯度权重控制器,动态优化路径级隐式奖励(SFT)与结果导向显式奖励(RL)的平衡。该方法正则化策略基于策略熵以保证稳定性,自主发现有效训练课程。在数学推理、抽象视觉推理(General Points)和视觉-语言导航(V-IRL)等多个挑战性基准上评估,AMFT持续取得新最优性能,并在分布外(OOD)任务上展现更强泛化能力。消融实验与训练动态分析证实,元学习控制器对模型稳定性、样本效率与性能至关重要,为大模型对齐提供了更合理有效的范式。代码已开源:https://github.com/hlxtsyj/AMFT。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are typically fine-tuned for reasoning tasks through a two-stage pipeline of Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL), a process fraught with catastrophic forgetting and suboptimal trade-offs between imitation and exploration. Recent single-stage methods attempt to unify SFT and RL using heuristics, but lack a principled mechanism for dynamically balancing the two paradigms. In this paper, we reframe this challenge through the theoretical lens of \textbf{implicit rewards}, viewing SFT and RL not as distinct methods but as complementary reward signals. We introduce \textbf{Adaptive Meta Fine-Tuning (AMFT)}, a novel single-stage algorithm that learns the optimal balance between SFT's implicit, path-level reward and RL's explicit, outcome-based reward. The core of AMFT is a \textbf{meta-gradient adaptive weight controller} that treats the SFT-RL balance as a learnable parameter, dynamically optimizing it to maximize long-term task performance. This forward-looking approach, regularized by policy entropy for stability, autonomously discovers an effective training curriculum. We conduct a comprehensive evaluation on challenging benchmarks spanning mathematical reasoning, abstract visual reasoning (General Points), and vision-language navigation (V-IRL). AMFT consistently establishes a new state-of-the-art and demonstrats superior generalization on out-of-distribution (OOD) tasks. Ablation studies and training dynamic analysis confirm that the meta-learning controller is crucial for AMFT's stability, sample efficiency, and performance, offering a more principled and effective paradigm for LLM alignment. Our codes are open-sourced via https://github.com/hlxtsyj/AMFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。