突破自回归局限，探索扩散模型等新范式提升代码生成与推理能力
Beyond the Autoregressive Horizon: A Comprehensive Survey of Diffusion Models, World Modelling, and State Space Models for Code
- 用扩散模型实现整体去噪生成，捕捉长程语法约束
- 引入代码世界模型模拟执行状态，增强程序语义理解
- 结合状态空间模型实现线性时间效率，支持超长上下文
自回归语言模型在自动化软件工程中推动了显著进展，实现了强大的代码生成与辅助系统。然而，逐词预测范式在代码推理上存在结构性局限，包括全局规划受限、长距离依赖难以维持以及对程序执行语义的支撑不足。鉴于现有文献高度偏向自回归模型，本文探讨了可能克服逻辑与扩展瓶颈的新兴范式，以释放下一代代码智能架构潜力。具体而言，讨论了扩散模型通过整体去噪生成代码，捕捉自回归模型常忽略的长程句法约束；代码世界模型（CWMs）通过模拟执行状态支持推理；状态空间模型（SSMs）则提供线性时间效率，适用于大规模上下文。结合认知神经科学发现，本文勾勒出构建“系统2”代码生成代理的发展方向。
原文摘要 · Abstract (English)
Autoregressive (AR) language models have driven significant progress in automated software engineering, enabling powerful code generation and assistance systems. However, the next-token prediction paradigm introduces structural limitations for code reasoning, including restricted global planning, challenges in maintaining long-range dependencies, and limited grounding in program execution semantics. Noting the heavy skewness of existing literature towards AR models, we discuss emerging paradigms that could potentially overcome the logic and scaling bottlenecks of next-token prediction by unlocking next-generation architectural capabilities for code intelligence. Specifically, we discuss the potential of Diffusion Models, which generate code via holistic denoising that captures long-range syntactic constraints often missed by AR models. We also discuss Code World Models (CWMs), which simulate execution states to support reasoning, and State Space Models (SSMs), which provide linear-time efficiency for massive contexts. By connecting these developments with findings from cognitive neuroscience, we outline directions for developing "System 2" code generation agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。