让小模型自主推理更准更强,还能避免依赖外部指导。
ReaLM: Reflection-Enhanced Autonomous Reasoning with Small Language Models
- 用正负路径对比提升推理能力
- 渐进式减弱外部信号,增强自主性
- 融合领域知识,提高泛化能力
小型语言模型(SLMs)在成本上优于大模型,但复杂推理时易出错且答案不一致。现有方法常以牺牲推理能力、自主性或泛化性为代价。本文提出ReaLM,一种强化学习框架,用于垂直领域的鲁棒自主推理。通过多路径过程验证(MRPV),对比正负推理路径提取关键模式,提升推理能力;引入渐进诱导自主性(EAAI)训练策略,逐步淡化外部信号,增强自主性;采用引导链式思维蒸馏,将领域规则与专家知识融入模型参数,提升泛化能力。在垂直与通用推理任务上的实验表明,ReaLM显著改善了三方面表现。
原文摘要 · Abstract (English)
Small Language Models (SLMs) are a cost-effective alternative to Large Language Models (LLMs), but often struggle with complex reasoning due to their limited capacity and a tendency to produce mistakes or inconsistent answers during multi-step reasoning. Existing efforts have improved SLM performance, but typically at the cost of one or more of three key aspects: (1) reasoning capability, due to biased supervision that filters out negative reasoning paths and limits learning from errors; (2) autonomy, due to over-reliance on externally generated reasoning signals; and (3) generalization, which suffers when models overfit to teacher-specific patterns. In this paper, we introduce ReaLM, a reinforcement learning framework for robust and self-sufficient reasoning in vertical domains. To enhance reasoning capability, we propose Multi-Route Process Verification (MRPV), which contrasts both positive and negative reasoning paths to extract decisive patterns. To reduce reliance on external guidance and improve autonomy, we introduce Enabling Autonomy via Asymptotic Induction (EAAI), a training strategy that gradually fades external signals. To improve generalization, we apply guided chain-of-thought distillation to encode domain-specific rules and expert knowledge into SLM parameters, making them part of what the model has learned. Extensive experiments on both vertical and general reasoning tasks demonstrate that ReaLM significantly improves SLM performance across aspects (1)-(3) above.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。