arXiv:2606.12373cs.CL2026-06被引 1

用可验证环境当积木,递归组合提升大模型推理泛化能力。

Verifiable Environments Are LEGO Bricks: Recursive Composition for Reasoning Generalization

论文配图:Verifiable Environments Are LEGO Bricks: Recursive Composition for Reasoning Generalization
图 1 · 摘自论文原文
  • 将可验证环境视为可递归组装的积木块,自动拼接形成新环境。
  • 在6个未见基准上,使模型平均性能提升3.1点,最高达2.3点。
  • 仅用50个基础环境即达300个环境的效果,大幅提高训练效率。

强化学习结合可验证环境已成为增强大语言模型推理能力的有效方法。尽管已有研究证明环境数量增加能提升强化学习性能,但传统手动或逐个构建方式存在线性扩展瓶颈,制约了可扩展的推理泛化。本文提出RACES(递归自动化组合用于环境扩展),将可验证环境视为可组合的积木块,实现递归式组装。核心思想是当一个环境的输出类型与另一个环境的输入类型匹配时,可自动融合生成新环境。RACES基于300个独立环境,设计了四种组合算子(SEQUENTIAL、PARALLEL、SORT、SELECT),诱导多样化的推理模式。大量实验表明,基于这些组合环境进行强化学习训练,显著提升了推理泛化能力:使DeepSeek-R1-Distill-Qwen-14B平均提升3.1分(从48.2升至51.3),Qwen3-14B在六项未见基准上从58.8提升至61.1。同时,仅用50个基础环境即达到300个环境的性能水平,证明其环境利用效率极高。

原文摘要 · Abstract (English)

Reinforcement Learning (RL) with verifiable environments has emerged as a powerful approach for enhancing the reasoning capabilities of Large Language Models (LLMs). While prior research demonstrates that scaling environment quantity improves RL performance, existing manual or individual construction methods suffer from linear scaling limits, thereby hindering scalable reasoning generalization. This paper introduces RACES (\textbf{R}ecursive \textbf{A}utomated \textbf{C}omposition for \textbf{E}nvironment \textbf{S}caling), a framework that conceptualizes verifiable environments as composable building blocks that can be recursively assembled. The key insight is that when the codomain (output type) of one environment matches the domain (input type) of another, they can be automatically fused into a new verifiable environment, enabling recursive composition. RACES is implemented with 300 individual environments and defines a set of composition operators (\textsc{SEQUENTIAL}, \textsc{PARALLEL}, \textsc{SORT}, and \textsc{SELECT}) that induce diverse reasoning patterns. Extensive experiments show that RL training on these composite environments consistently enhances reasoning generalization. Specifically, RACES improves DeepSeek-R1-Distill-Qwen-14B by an average of 3.1 points (from 48.2 to 51.3) and boosts Qwen3-14B performance from 58.8 to 61.1 on six benchmarks, which are unseen during the construction of training environments. Moreover, RACES achieves performance comparable to training on 300 individual environments using only 50 base environments, demonstrating significant efficiency in environment utilization.

强化学习推理泛化环境组合大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。