arXiv:2608.30922cs.AI2026-08

让扩散语言模型自动伸缩生成长度,又准又省算力。

CARVE: Verified Expansion for Variable-Length Generation in Diffusion Language Models

论文配图:CARVE: Verified Expansion for Variable-Length Generation in Diffusion Language Models
图 1 · 摘自论文原文
  • 先用短掩码框生成,再按需动态插入新掩码位。
  • 通过反事实检验决定是否保留新增掩码,降低错误风险。
  • 无需重训练,通用性强,推理成本减半且准确率提升。

掩码扩散语言模型通过部分可见的响应画布预测词元,支持双向条件与并行优化。然而,标准掩码扩散解码器采用固定长度接口:生成前即确定答案掩码位置数量,选择困难——过短会截断推理或代码,过长浪费计算且干扰去噪。我们提出CARVE(反事实感知揭示与验证扩展),一种无需重训练的可变长度算法,适用于掩码扩散语言模型。从较短画布开始,解码过程中可动态插入额外[掩码]位置。不直接保留所有插入,而是测试候选扩展画布,并提出反事实问题:若存在额外掩码空间,模型对原未解位置的预测是否仍相似?仅当对齐未解位置的杰恩-申诺(JS)散度较低时才保留插入。使长度扩展成为可验证的稳定性决策,而非单纯置信度判断。CARVE无需重训练即可应用于全画布与分块扩散解码器。在代码生成与数学推理基准上,其平均性能超越所有评估模型家族的固定长度基线。关键优势在于,在获得更高准确率的同时显著降低推理成本,部分场景下仅需固定长度解码一半的浮点运算量(FLOPs)。

原文摘要 · Abstract (English)

Masked diffusion language models predict tokens from a partially observed response canvas, enabling bidirectional conditioning and parallel token refinement. Yet standard masked-diffusion decoders use a rigid inference interface: the number of masked positions allocated to the answer is fixed before generation begins. Choosing this length is difficult. A short canvas can truncate reasoning or code, while a long canvas wastes computation and can perturb denoising. We introduce CARVE (Counterfactual-Aware Reveal with Verified Expansion), a training-free variable-length algorithm for masked diffusion LMs. Starting from a shorter canvas, CARVE can grow the response during decoding by inserting additional [MASK] positions. Rather than keeping every insertion, CARVE tests a candidate expanded canvas and asks a counterfactual question: would the model make similar predictions for the unresolved positions in the original canvas if the extra masked space were present? The inserted masks are kept only when they induce low Jensen-Shannon (JS) divergence on aligned unresolved positions. This makes length growth a verified stability decision rather than a pure confidence heuristic. CARVE applies without retraining to both full-canvas and blockwise diffusion decoders. Across code generation and mathematical reasoning benchmarks, CARVE consistently improves average performance over fixed-length baselines across all evaluated model families. Crucially, CARVE achieves these accuracy gains while reducing inference cost, reaching half the FLOPs of fixed-length decoding in some settings.

扩散模型生成长度推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。