动态调整采样权重，让模型更关注影响生成效率的关键位置。
D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Drafting
- 根据位置对接受长度的梯度贡献动态分配训练权重
- 在6个基准上平均提升2.3%的推理速度和生成长度
- 无需修改模型结构或推理流程，仅需替换损失函数
推测解码通过小型草案模型并行提出候选词，由大型目标模型验证以加速大语言模型推理。近期基于扩散的并行草案模型（如DFlash）可在一次前向传播中预测完整的B-token块，实现更深的草案模型和更长的接受块。然而，现有多标记草案模型的目标函数通常采用固定的位置依赖加权策略（如头依赖权重或块位置衰减），无法随训练过程中限制接受率的位置变化而自适应调整。为此，本文从可微分的预期接受草案长度代理中推导出各位置的训练权重，使每个位置的权重与其对日志概率梯度的贡献相匹配。由此提出的损失函数D-PACE（动态位置感知交叉熵）能将训练信号自动聚焦于当前限制接受率的位置，随着草案模型性能提升而动态转移。在六个基准、两个Qwen3-4B草案深度、两种解码温度及两种额外目标模型上，D-PACE持续提升了墙钟速度和平均输出长度，训练时间仅增加2.3%，且无需改变草案模型架构或推理过程。
原文摘要 · Abstract (English)
Speculative decoding accelerates LLM inference by having a small drafter propose tokens that a larger target model verifies in parallel. Recent diffusion-based parallel drafters such as DFlash predict the full B-token block in one forward pass, enabling deeper drafters and longer accepted blocks. However, existing multi-token drafter objectives often use fixed position-dependent weighting schedules, such as head-dependent weights or block-position decays, which do not adapt as the positions limiting acceptance change during training. To address this, we derive per-position training weights from a differentiable surrogate of expected accepted draft length, matching the weight of each position to its log-probability gradient contribution. The resulting loss, D-PACE (Dynamic Position-Aware Cross-Entropy), shifts training signal toward positions that currently limit acceptance as the drafter improves. Across six benchmarks, two Qwen3-4B draft depths, two decoding temperatures, and two additional target models, D-PACE consistently improves both wall-clock speedup and average emitted length, with 2.3\% measured training-time overhead and no changes to the drafter architecture or inference procedure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。