arXiv:2507.17307cs.LGcs.AI2025-07被引 7

用熵值动态分配推理任务,让小模型高效处理简单步骤,大幅提速大模型推理。

R-Stitch: Dynamic Trajectory Stitching for Efficient Reasoning

  • 根据分词熵值判断难易,低熵由小模型处理,高熵交由大模型决策。
  • 在多个大模型上实现3.85倍至4.10倍加速,准确率几乎无损。
  • 无需训练即可适配不同算力需求,适合部署在资源受限场景。

链式思维(CoT)虽提升大语言模型(LLM)的解题能力,但因生成长序列导致推理开销巨大。现有加速方法或通过早停/压缩缩短轨迹,或采用小模型推测解码,但后者在模型意见不一致时增益有限,且强制逐标记一致,忽视部分小模型正确时可生成更简洁推理路径的事实。本文提出R-Stitch,一种无需训练的混合解码框架,利用分词熵作为不确定性代理,动态分配计算任务:小语言模型(SLM)处理低熵标记,大模型仅介入高熵不确定情况,避免全回滚并维持答案质量。进一步推出R-Stitch⁺,通过学习自适应路由策略,动态调整令牌预算,突破固定阈值限制。该方法同时降低每标记解码复杂度与总生成标记数,在保持接近全量大模型精度的前提下,实现显著加速——在DeepSeek-R1-Distill-Qwen-7B上达3.00×,14B上达3.85×,QWQ-32B上达4.10×。且天然支持按需调节效率-精度平衡,适用于多样计算预算,无需重新训练。

原文摘要 · Abstract (English)

Chain-of-thought (CoT) enhances the problem-solving ability of large language models (LLMs) but incurs substantial inference cost due to long autoregressive trajectories. Existing acceleration strategies either shorten traces via early stopping or compression, or adopt speculative decoding with a smaller model. However, speculative decoding provides limited gains when model agreement is low and rigidly enforces token-level consistency, overlooking the observation that some smaller models, when correct, produce significantly more concise reasoning traces that could reduce inference length. We introduce R-Stitch, a training-free hybrid decoding framework that leverages token-level entropy as an uncertainty proxy to delegate computation between a small language model (SLM) and an LLM. Our analysis shows that high-entropy tokens are more likely to induce errors, motivating an entropy-guided routing strategy that lets the SLM efficiently handle low-entropy tokens while delegating uncertain ones to the LLM, thereby avoiding full rollbacks and preserving answer quality. We further extend this design with R-Stitch$^{+}$, which learns an adaptive routing policy to adjust the token budget dynamically beyond fixed thresholds. By jointly reducing per-token decoding complexity and the number of generated tokens, our method achieves substantial acceleration with negligible accuracy loss. Concretely, it attains peak speedups of 3.00$\times$ on DeepSeek-R1-Distill-Qwen-7B, 3.85$\times$ on 14B, and 4.10$\times$ on QWQ-32B while maintaining accuracy comparable to full LLM decoding. Moreover, it naturally enables adaptive efficiency--accuracy trade-offs that can be tailored to diverse computational budgets without retraining.

推理加速混合解码大模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。