让大模型通过自我评估学会深度推理,无需人工标注奖励。
ALIVE: Awakening LLM Reasoning via Adversarial Learning and Instructive Verbal Evaluation
- 用对抗学习+语言反馈,让模型从文本中自建判断标准。
- 数学、编程、逻辑任务准确率提升,跨领域泛化更强。
- 适合想构建自主推理能力的研究者和开发者。
大语言模型实现专家级推理面临持续的‘奖励瓶颈’:传统强化学习依赖成本高、易失效且忽视解题逻辑的标量奖励信号,阻碍模型形成内在的推理理解。我们提出ALIVE(对抗学习与指导性语言评价),一种无需人工干预的对齐框架,将问题生成、求解与评判统一于单一策略模型中,内化正确性的逻辑。通过对抗学习结合指导性语言反馈,模型可直接从原始语料中习得评估标准,将外部批评转化为内生推理能力。在数学推理、代码生成和通用逻辑推理基准上的实验证明,相同数据与算力下,ALIVE显著缓解了奖励信号局限,实现准确率提升、更强跨域泛化及更高自我修正率。结果表明,推理三要素协同促进能力持续增长,使ALIVE成为无需人工监督的通用推理对齐可扩展基础。
原文摘要 · Abstract (English)
The quest for expert-level reasoning in Large Language Models (LLMs) has been hampered by a persistent \textit{reward bottleneck}: traditional reinforcement learning (RL) relies on scalar rewards that are \textbf{costly} to scale, \textbf{brittle} across domains, and \textbf{blind} to the underlying logic of a solution. This reliance on external, impoverished signals prevents models from developing a deep, self-contained understanding of reasoning principles. We introduce \textbf{ALIVE} (\emph{Adversarial Learning with Instructive Verbal Evaluation}), a hands-free alignment framework that moves beyond scalar reward optimization toward intrinsic reasoning acquisition. Grounded in the principle of \emph{Cognitive Synergy}, ALIVE unifies problem posing, solving, and judging within a single policy model to internalize the logic of correctness. By coupling adversarial learning with instructive verbal feedback, ALIVE enables models to internalize evaluative criteria directly from raw corpora, effectively transforming external critiques into an endogenous reasoning faculty. Empirical evaluations across mathematical reasoning, code generation, and general logical inference benchmarks demonstrate that ALIVE consistently mitigates reward signal limitations. With identical data and compute, it achieves accuracy gains, markedly improved cross-domain generalization, and higher self-correction rates. These results indicate that the reasoning trinity fosters a self-sustaining trajectory of capability growth, positioning ALIVE as a scalable foundation for general-purpose reasoning alignment without human-in-the-loop supervision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。