用强化学习提升大模型通用推理能力,更高效且结果更简洁。
Apriel-1.5-OpenReasoner: RL Post-Training for General-Purpose and Efficient Reasoning

- 通过自适应采样和难度感知惩罚,跨领域优化推理过程。
- 在多项评测中超越基线模型,推理路径缩短30%-50%。
- 适合追求高效精准推理的开发者与研究者使用。
利用可验证奖励的强化学习(RLVR)构建通用推理模型已成为前沿开源大模型的常见方法,但其训练方案和领域混合策略常未公开。跨领域联合优化面临挑战:各领域推演长度、问题难度和样本效率差异大。此外,长链式推理增加推理开销与延迟,效率至关重要。本文提出 Apriel-1.5-OpenReasoner,基于 150 亿参数开源模型 Apriel-Base,采用完全可复现的多领域强化后训练方案,在数学、代码生成、指令遵循、逻辑谜题和函数调用五个领域使用公开数据集进行训练。引入自适应领域采样机制,确保目标领域比例,即使在推演动态异质情况下仍有效;设计难度感知的长度惩罚扩展,无需额外训练成本即可鼓励困难问题生成更长推理,简单问题生成更短路径。在严格 16K token 输出预算下训练,推理时可扩展至 32K tokens,优于 Apriel-Base 在 AIME 2025、GPQA、MMLU-Pro 与 LiveCodeBench 上的表现,同时推理轨迹减少 30%-50%。在相近规模模型中达到更高精度-令牌预算权衡,推动了性能边界。
原文摘要 · Abstract (English)
Building general-purpose reasoning models using reinforcement learning with verifiable rewards (RLVR) across diverse domains has been widely adopted by frontier open-weight models. However, their training recipes and domain mixtures are often not disclosed. Joint optimization across domains poses significant challenges: domains vary widely in rollout length, problem difficulty and sample efficiency. Further, models with long chain-of-thought traces increase inference cost and latency, making efficiency critical for practical deployment. We present Apriel-1.5-OpenReasoner, trained with a fully reproducible multi-domain RL post-training recipe on Apriel-Base, a 15B-parameter open-weight LLM, across five domains using public datasets: mathematics, code generation, instruction following, logical puzzles and function calling. We introduce an adaptive domain sampling mechanism that preserves target domain ratios despite heterogeneous rollout dynamics, and a difficulty-aware extension of the standard length penalty that, with no additional training overhead, encourages longer reasoning for difficult problems and shorter traces for easy ones. Trained with a strict 16K-token output budget, Apriel-1.5-OpenReasoner generalizes to 32K tokens at inference and improves over Apriel-Base on AIME 2025, GPQA, MMLU-Pro, and LiveCodeBench while producing 30-50% shorter reasoning traces. It matches strong open-weight models of similar size at lower token cost, thereby pushing the Pareto frontier of accuracy versus token budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。