用模型自动生成推理路径,能省下大量人工标注成本。
Reasoning Pattern Matters: Learning to Reason without Human Rationales
- 让大模型自动学习固定推理模式来生成答案
- 用模型生成的推理路径,效果接近人工标注10倍数据
- 适合需要大量推理训练但标注难的任务
大型语言模型在广泛采用的SFT+RLVR范式下展现出强大推理能力:先用人类标注的推理轨迹(理由)进行监督微调(SFT),建立初始推理行为;再通过可验证奖励的强化学习(RLVR)优化模型,无需黄金标准理由。然而,高质量理由的人工标注成本极高。本文研究发现,在一类称为‘模式化推理任务’的问题中,尽管实例内容各异(如领域知识、事实信息或数值),但解题均依赖一致的程序性策略。我们证明,SFT+RLVR的成功主要源于模型对这些推理模式的内化。以数值语义匹配为例,通过因果与行为证据表明,推理模式比理由的数量或质量更关键。基于此,提出PARO框架——让模型在无需人工理由标注的前提下,生成符合任务特定推理模式的理由。实验显示,PARO生成的理由在SFT+RLVR中表现媲美人工标注且规模达10倍的数据。结果表明,大规模人工理由标注可被仅需有限人类监督的模型自动标注替代。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable reasoning capabilities under the widely adopted SFT+RLVR paradigm, which first performs Supervised Fine-Tuning (SFT) on human-annotated reasoning trajectories (rationales) to establish initial reasoning behaviors, then applies Reinforcement Learning with Verifiable Rewards (RLVR) to optimize the model using verifiable signals without golden rationales. However, annotating high-quality rationales for the SFT stage remains prohibitively expensive. This paper investigates when and how rationale annotation costs can be substantially reduced without compromising reasoning performance. We identify a broad class of problems, termed patterned reasoning tasks, where reasoning follows a fixed, procedural strategy consistent across instances. Although instances vary in content such as domain knowledge, factual information, or numeric values, the solution derives from applying a shared reasoning pattern. We argue that the success of SFT+RLVR on such tasks primarily stems from its ability to enable models to internalize these reasoning patterns. Using numerical semantic matching as a representative task, we provide both causal and behavioral evidence showing that reasoning patterns rather than the quantity or quality of rationales are the key determinant of performance. Building on these insights, we propose Pattern-Aware LLMs as Rationale AnnOtators (PARO), a simple yet effective framework that enables LLMs to generate rationales aligned with task-specific reasoning patterns without requiring human rationale annotations. Experiments show that PARO-generated rationales achieve comparable SFT+RLVR performance to human rationales that are 10 times larger. These results suggest that large-scale human rationale annotations can be replaced with LLM-based automatic annotations requiring only limited human supervision over reasoning patterns.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。