arXiv:2606.26744cs.LGcs.CL2026-06

针对DeepSeek-V4的超连接结构优化了并行推测解码,提升生成速度与准确性。

HyperDFlash: Hyper-Connection-Aligned Block Speculative Decoding with Gated Residual Reduction

论文配图:HyperDFlash: Hyper-Connection-Aligned Block Speculative Decoding with Gated Residual Reduction
图 1 · 摘自论文原文
  • 用预坍缩残差状态作为唯一条件信号,保留多路径结构信息。
  • 轻量门控残差压缩器减少参数量三个数量级,保持架构对齐。
  • 通过目标分布正则化训练,提升早期草稿质量,适合大模型高效推理。

我们提出HyperDFlash,一种专为DeepSeek-V4的超连接(HC)结构设计的块并行推测解码框架。尽管DeepSeek-V4原生的多标记预测(MTP)模块在初始标记起草中表现良好,但随着生成位置后移,其草稿准确率急剧下降,因未验证中间标记的误差累积导致草稿接受率降低。尽管原始DFlash方法支持高效的单次遍历块起草,却难以无缝适配HC范式,因DeepSeek-V4的多路径残差流与传统起草设计存在固有特征错位。为此,我们提出两项面向模型的专用优化:首先,采用预坍缩残差状态作为唯一条件信号,完整保留多路径结构信息,更好对齐目标模型的原生预测路径;其次,以轻量门控残差缩减器替代重型通用线性压缩器,其参数直接继承自目标模型内置的hc_head模块。该设计实现输入感知路径聚合,参数量减少三个数量级,同时保持精确架构对齐。此外,通过在LM头施加针对性的KL蒸馏损失,对模型训练进行增强,使预测结果更贴近目标分布,改善早期草稿质量。在数学推理、代码生成和对话基准上的大量实验表明,HyperDFlash持续优于原生MTP基线与普通DFlash适配版本,在平均接受草稿长度和解码加速比上均取得显著提升。结果验证了HC对齐、门控缩减和针对性蒸馏对高性能推测解码的有效性。

原文摘要 · Abstract (English)

We present HyperDFlash, a block-parallel speculative decoding framework tailored to DeepSeek-V4's Hyper-Connections (HC). Despite the strong performance of DeepSeek-V4's native Multi-Token Prediction (MTP) module on initial token drafting, its draft accuracy degrades sharply at later positions, as error accumulation from unverified intermediate tokens harms draft acceptance rates. Although the original DFlash method supports efficient one-pass block drafting, it cannot be seamlessly adapted to the HC paradigm, since DeepSeek-V4's multi-path residual stream induces inherent feature misalignment with conventional drafting designs. To resolve this architectural mismatch, we propose two dedicated, model-aligned optimizations for HC residual streams. First, we adopt pre-collapse residual states as the exclusive conditioning signal, preserving complete multi-path structural information and better aligning the drafter with the target's native prediction pathway. Second, we replace the heavy generic linear compressor with a lightweight gated residual reducer, whose parameters are directly inherited from the target model's built-in hc_head module. This design yields input-aware path aggregation with three orders of magnitude fewer parameters while maintaining precise architectural alignment. We further enhance model training via a targeted KL distillation loss applied to the LM-head, regularizing predictions against the target distribution to improve early draft quality. Extensive experiments across math reasoning, code synthesis, and conversational benchmarks demonstrate that HyperDFlash consistently outperforms both the native MTP baseline and vanilla DFlash adaptation, achieving substantial gains in average accepted draft length and decoding speedup. These results validate HC alignment, gated reduction, and targeted distillation for high-performance speculative decoding.

推测解码大模型推理架构对齐DeepSeek-V4

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。