强化学习能否合成新技能？关键在于先掌握基础能力。
Atomic Skills are the Prerequisite: When Reinforcement Learning Synthesizes Compositional Reasoning, and When It Only Amplifies
- 将复杂推理拆解为参数化与情境化两种基础技能
- 仅靠监督微调在新信息上准确率暴跌至18%
- 强化学习可真正合成新策略，但需先掌握基础技能
强化学习（RL）是单纯放大已有技能，还是能合成新技能？本文通过互补推理这一关键能力——将内部知识与外部上下文结合——来回答此问题。为避免预训练干扰，我们构建了一个生物信息语义合成数据集，并将其分解为两种原子技能：参数化推理（从模型权重中检索事实）和情境化推理（处理上下文中的新信息）。实验发现：直接在复合任务上监督微调的模型，在已见事实和推理路径上准确率达90%，但在新事实和推理路径上骤降至18%，表明其依赖机械记忆而非真实技能融合；而强化学习能弥合这一泛化鸿沟，成为真正的技能合成器——但前提是基础模型必须先通过监督微调掌握独立的原子技能。结果表明，先分步训练原子技能再进行强化学习，是一条通往复杂新推理的可扩展路径。
原文摘要 · Abstract (English)
Does Reinforcement Learning (RL) merely amplify existing skills, or synthesize novel skills? We investigate this question through the lens of Complementary Reasoning: the critical practical capability of integrating internal knowledge with external context, a prerequisite for reliable Continual Learning and Retrieval-Augmented Generation. To avoid pre-training contamination, we construct a controlled semanticsynthetic dataset of biographies and decompose this capability into two atomic skills: Parametric Reasoning (retrieving facts encoded in model weights) and Contextual Reasoning (processing novel in-context information). We present two findings. First, models supervised directly on the composite task reach high accuracy on seen facts and reasoning paths (90%) but collapse on novel facts and reasoning paths (18%), indicating that Supervised Fine-Tuning (SFT) relies on rote memorization rather than genuine skill integration. Second, RL bridges this generalization gap, acting as a skill synthesizer rather than a mere amplifier--but only under a strict prerequisite: it synthesizes new composite strategies only when the base model has first mastered the independent atomic skills via SFT. These results suggest that decoupled atomic training followed by RL offers a scalable path to complex novel reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。