让复杂推理更快更透明，通过并行生成子步骤实现高效可解释强化学习。
DeCoRL: Decoupling Reasoning Chains via Parallel Sub-Step Generation and Cascaded Reinforcement for Interpretable and Scalable RLHF
- 并行生成推理子步骤，打破传统串行瓶颈。
- 推理速度提升3.8倍，能耗降低72.4%，吞吐量提高68%。
- 每步独立评分，错误定位更清晰，适合高可靠性场景。
现有基于思维链的强化学习方法存在两大局限：一是作为黑箱整体输出奖励，无法区分各步骤贡献，难以诊断错误；二是串行解码导致时间复杂度为O(n)，难以实现实时部署。本文提出DeCoRL（通过协同强化学习解耦推理链），将推理从串行处理转变为模块化协作。该框架训练轻量级专用模型并行生成推理子步骤，消除串行瓶颈。设计模块化奖励函数对每个子步骤独立评分，实现精准错误归因。通过级联式DRPO优化协调各步奖励，同时保持步骤间依赖关系。综合评估显示，在RM-Bench、RMB和RewardBench上均达到领先性能，优于现有方法，包括大模型。该方法实现3.8倍加速，能耗降低72.4%，吞吐量提升68%，且在可解释性上提升22.7%。这些进展使复杂推理系统的实时部署成为可能。
原文摘要 · Abstract (English)
Existing reinforcement learning methods for Chain-of-Thought reasoning suffer from two critical limitations. First, they operate as monolithic black boxes that provide undifferentiated reward signals, obscuring individual step contributions and hindering error diagnosis. Second, sequential decoding has O(n) time complexity. This makes real-time deployment impractical for complex reasoning tasks. We present DeCoRL (Decoupled Reasoning Chains via Coordinated Reinforcement Learning), a novel framework that transforms reasoning from sequential processing into collaborative modular orchestration. DeCoRL trains lightweight specialized models to generate reasoning sub-steps concurrently, eliminating sequential bottlenecks through parallel processing. To enable precise error attribution, the framework designs modular reward functions that score each sub-step independently. Cascaded DRPO optimization then coordinates these rewards while preserving inter-step dependencies. Comprehensive evaluation demonstrates state-of-the-art results across RM-Bench, RMB, and RewardBench, outperforming existing methods including large-scale models. DeCoRL delivers 3.8 times faster inference while maintaining superior solution quality and offers a 22.7\% improvement in interpretability through explicit reward attribution. These advancements, combined with a 72.4\% reduction in energy consumption and a 68\% increase in throughput, make real-time deployment of complex reasoning systems a reality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。