让大模型学会从错误中提炼经验,提升推理能力。
Internalizing Meta-Experience into Memory for Guided Reinforcement Learning in Large Language Models
- 通过对比正确与错误推理路径,定位出错关键节点。
- 将错误经验转化为可复用的元经验,提升模型记忆。
- 适用于需要精准推理的大模型任务,如数学和逻辑题。
基于可验证奖励的强化学习(RLVR)已成为提升大语言模型(LLM)推理能力的有效方法。然而,RLVR存在元学习瓶颈:缺乏人类学习周期中误差归因与经验内化的机制,限制了细粒度信用分配和可复用知识的形成。本文将此类源于过往错误的可复用知识表示为元经验。为此,提出元经验学习(MEL)框架,将自蒸馏的元经验引入模型参数化记忆中。在标准RLVR基础上,利用LLM的自验证能力,对成对的正确与错误轨迹进行对比分析,精准识别推理错误发生的关键分叉点,并将其总结为可泛化的元经验。该元经验通过最小化负对数似然内化至LLM的参数记忆,生成语言建模的奖励信号,连接正确与错误推理路径,促进知识有效重用。实验表明,MEL在多个基准上均取得一致提升,不同模型规模下Pass@1指标提高3.92%至4.73%。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach for enhancing the reasoning capabilities of Large Language Models (LLMs). Despite its efficacy, RLVR faces a meta-learning bottleneck: it lacks mechanisms for error attribution and experience internalization intrinsic to the human learning cycle beyond practice and verification, thereby limiting fine-grained credit assignment and reusable knowledge formation. We term such reusable knowledge representations derived from past errors as meta-experience. Based on this insight, we propose Meta-Experience Learning (MEL), a novel framework that incorporates self-distilled meta-experience into the model's parametric memory. Building upon standard RLVR, we introduce an additional design that leverages the LLM's self-verification capability to conduct contrastive analysis on paired correct and incorrect trajectories, identify the precise bifurcation points where reasoning errors arise, and summarize them into generalizable meta-experience. The meta-experience is further internalized into the LLM's parametric memory by minimizing the negative log-likelihood, which induces a language-modeled reward signal that bridges correct and incorrect reasoning trajectories and facilitates effective knowledge reuse. Experimental results demonstrate that MEL achieves consistent improvements on benchmarks, yielding 3.92%--4.73% Pass@1 gains across varying model sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。