用强化学习优化推理草稿，提升多模态模型的生成速度与准确性。
DREAM-R: Multimodal Speculative Reasoning with RL-Based Refined Drafting, Precise Verification, and Fully Parallel Execution
- 通过强化学习训练草稿模型生成更贴近目标推理路径的内容。
- 采用比率阈值验证机制，仅在证据充分时接受草稿，避免错误传播。
- 支持多步并行推理，实现早期终止和干净回退，显著提速且不丢精度。
推测性推理被提出用于加速大型多模态模型中的高推理强度生成任务，但其效果常受限于草稿与目标验证推理之间的不一致。本文提出 DREAM-R 框架，显著提升推测性推理性能。核心在于引入基于强化学习的推测对齐策略优化（SAPO），训练草稿模型生成既忠实于目标轨迹又简洁的推理步骤。进一步提出基于阈值的验证机制（TBVM），采用比率标准，在正向证据明显占优时才接受推测步骤，确保验证稳定且可解释。在此基础上构建全并行推测推理（FPSR）框架，实现草稿生成、目标侧推理与验证在多步推理中的并行执行，支持早期终止与干净回退。在多个高推理强度基准测试中，实验表明该方法实现高达数倍的速度提升，同时保持目标模型精度，带来显著效率增益而不牺牲推理质量。
原文摘要 · Abstract (English)
Speculative reasoning has recently been proposed as a means to accelerate reasoning-intensive generation in large multimodal models, but its effectiveness is often constrained by misalignment between speculative drafts and target-verified reasoning. In this work, we introduce DREAM-R, a framework that substantially improves the performance of speculative reasoning. At its core, DREAM-R employs Speculative Alignment Policy Optimization (SAPO), a reinforcement-learning objective that trains draft models to generate reasoning steps that are both faithful to target trajectories and concise. We further propose a Threshold-based Verification Mechanism (TBVM) that uses a ratio-based criterion to provide stable and interpretable acceptance of speculative steps only when positive evidence clearly dominates, thereby preventing error propagation. Building on these components, we develop a Fully Parallel Speculative Reasoning (FPSR) framework that parallelizes draft generation, target-side reasoning, and verification across multi-step reasoning, enabling early stopping and clean fallback. Experiments on reasoning-heavy benchmarks demonstrate up to speedup while preserving target-model accuracy, yielding substantial efficiency gains without compromising reasoning quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。