arXiv:2607.25852cs.CL2026-07被引 1

提出统一框架,让大模型推理速度提升近2.5倍且自适应不同任务。

AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding

论文配图:AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
图 1 · 摘自论文原文
  • 分场景训练两种生成器:聊天用轻量预测,代码数学用长序列扩散。
  • 新架构DFly提升并行效率,验证阶段动态分配算力,最高提速2.4倍。
  • 适合高并发、多任务的生产环境,尤其对代码和长文本生成优化显著。

推测解码可加速大语言模型推理而不改变输出分布,但单一生成结构难以应对真实负载。自回归多标记预测(MTP)轻量稳定,块并行扩散能摊薄生成延迟,最优选择取决于输出分布。我们提出AngelSpec,一个统一训练框架,从三方面解决异构性:训练层面,将MTP drafter用于多样化对话数据以应对高熵开放问答,将块扩散drafter用于代码与数学数据以生成长而可预测的延续;架构层面,提出DFly,结合混合目标条件主干与前驱条件自回归头,提升目标特征利用与块内依赖建模能力,同时保持生成并行;推理层面,根据领域、请求、在线负载和硬件动态调整验证深度,将验证视为共享批处理资源,优先分配高置信前缀计算。在Hy3系列测试中,DFly使Hy3-A21B平均接受长度提升约30%,在并发数4至64下均达到最高吞吐量,比自回归解码快1.98-2.40倍,比DFlash高10.5-11.8%。代码与模型已开源支持扩展。

原文摘要 · Abstract (English)

Speculative decoding accelerates large language model inference without changing the target distribution, but no single drafting structure performs best across real-world workloads. Autoregressive multi-token prediction (MTP) is a lightweight, stable proposal mechanism, whereas block-parallel diffusion amortizes drafting latency over much longer candidate sequences; the better choice depends strongly on the output distribution. We present AngelSpec, a unified training framework for MTP and block-parallel speculative decoding that addresses this heterogeneity at three levels. At the training level, rather than fitting one universal drafter to a uniform data mixture, we co-specialize structure and data: the MTP drafter is trained on diverse conversational data for high-entropy open-ended chat, and the block-diffusion drafter on code and mathematics data for longer predictable continuations. At the architecture level, we propose DFly, a block-diffusion framework combining a hybrid target-conditioning backbone with a predecessor-conditioned autoregressive head, improving target-feature utilization and intra-block dependency modeling while keeping generation parallel. At the inference level, both acceptance length and verification cost vary with domain, request, online load, and hardware, so DFly treats verification as a shared batch-level resource: it reallocates compute toward high-confidence prefixes across requests and combines expected utility with a profiled cost model to adapt verification depth online. Across the Hy3 series, DFly raises the average accepted length on Hy3-A21B by roughly 30% and attains the highest average throughput at every tested concurrency from 4 to 64, a 1.98-2.40x speedup over autoregressive decoding and 10.5-11.8% higher throughput than DFlash. We release AngelSpec to support training and extending these methods.

推理加速推测解码大模型自适应

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。