让扩散语言模型在生成中灵活插入提示，提速近4倍且更准。
Thinking Inside the Mask: In-Place Prompting in Diffusion LLMs
- 在生成过程中直接向掩码位置插入提示，实现双向信息交互。
- 在GSM8K上准确率提升17.29%，速度加快4.12倍；MMLU加速276.67倍。
- 通过置信度判断提前退出，大幅降低计算开销，适合高效推理场景。
尽管大语言模型（LLMs）取得了显著成功，但其仅支持前缀提示和串行生成的范式限制了双向信息流动。扩散大语言模型（dLLMs）凭借双向注意力机制和迭代优化过程，为更灵活的在位提示策略提供了新可能。我们提出ICE（在位思维链提示与早期退出），一种专为dLLMs设计的新框架，将前缀提示转化为在位提示，在迭代优化过程中将提示直接嵌入掩码位置，并引入基于置信度的早期退出机制，显著减少计算开销。大量实验表明，ICE在GSM8K上实现最高17.29%的准确率提升和4.12×的速度加速，在MMLU上实现高达276.67×的加速，同时保持竞争力表现。
原文摘要 · Abstract (English)
Despite large language models (LLMs) have achieved remarkable success, their prefix-only prompting paradigm and sequential generation process offer limited flexibility for bidirectional information. Diffusion large language models (dLLMs) present new opportunities through their bidirectional attention mechanisms and iterative refinement processes, enabling more flexible in-place prompting strategies. We introduce ICE (In-Place Chain-of-Thought Prompting with Early Exit), a novel framework that transforms prefix-only prompting into in-place prompting specifically designed for dLLMs. ICE integrates in-place prompts directly within masked token positions during iterative refinement and employs a confidence-aware early exit mechanism to significantly reduce computational overhead. Extensive experiments demonstrate ICE's effectiveness, achieving up to 17.29% accuracy improvement with 4.12$\times$ speedup on GSM8K, and up to 276.67$\times$ acceleration on MMLU while maintaining competitive performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。