解决扩散语言模型生成时因上下文不完整导致的错误传播问题
CORE: Context-Robust Remasking for Diffusion Language Models
- 通过检测令牌对上下文扰动的敏感度识别易错项
- 在推理阶段实现无需训练的修复，代码生成提升9.2个百分点
- 特别适合需要高准确率的推理与编程任务
标准掩码扩散模型解码受上下文僵化限制：令牌保留基于瞬时高置信度，常忽略早期预测缺乏完整上下文的事实。这引发级联效应，初始错误误导后续生成。现有修正策略依赖静态置信度，但该信号本质短视；不一致的令牌可能对模型自身表现出高置信。我们提出无需训练的推理时修正框架CORE，不依赖静态概率，而是通过探测目标掩码上下文扰动下的敏感性来识别上下文脆弱令牌。将修正形式化为对上下文变化的鲁棒优化目标，并高效近似以优先处理不稳定令牌。在LLaDA-8B-Base上，CORE在推理与代码基准上均取得稳定提升，超越计算量相当的基线，MBPP提升达9.2个百分点。
原文摘要 · Abstract (English)
Standard decoding in Masked Diffusion Models (MDMs) is hindered by context rigidity: tokens are retained based on transient high confidence, often ignoring that early predictions lack full context. This creates cascade effects where initial inconsistencies misguide the remaining generation. Existing revision strategies attempt to mitigate this by relying on static confidence scores, but these signals are inherently myopic; inconsistent tokens can appear confident to the model itself. We propose Context-Robust Remasking (CORE), a training-free framework for inference-time revision. Rather than trusting static token probabilities, CORE identifies context-brittle tokens by probing their sensitivity to targeted masked-context perturbations. We formalize revision as a robust optimization objective over context shifts and efficiently approximate this objective to prioritize unstable tokens for revision. On LLaDA-8B-Base, CORE delivers consistent improvements across reasoning and code benchmarks, outperforming compute-matched baselines and improving MBPP by up to 9.2 percentage points.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。