无需外部验证器的扩散模型,通过内在能量函数实现可扩展推理
VFScale: Intrinsic Reasoning through Verifier-Free Test-time Scalable Diffusion Model
- 用扩散模型自身能量函数替代外部验证器,实现内在推理
- 在15×15迷宫中解决率88%,远超传统扩散模型
- 结合新型混合蒙特卡洛树搜索,提升推理效率
受人类系统2思维启发,大语言模型通过链式思维在复杂推理任务中表现优异。然而,扩散模型在测试时的可扩展推理仍缺乏探索。现有方法存在两大挑战:(i)依赖外部验证器,偏离人类智能的内在推理;(ii)缺乏高效搜索算法。本文提出无验证器测试时可扩展扩散模型(VFScale),利用扩散模型自身的内在能量函数作为验证机制,实现可扩展的内在推理。训练方面,引入新型MRNCL损失与KL正则化,优化能量分布,使能量函数本身具备可靠验证能力;推理方面,将去噪过程与新型混合蒙特卡洛树搜索(hMCTS)融合,提升搜索效率。在迷宫和数独等复杂推理任务中,使用最大6×6迷宫训练的VFScale,在15×15迷宫上解决了88%的问题,而标准扩散模型完全失败。
原文摘要 · Abstract (English)
Inspired by human SYSTEM 2 thinking, LLMs excel at complex reasoning tasks via extended Chain-of-Thought. However, similar test-time scaling for diffusion models to tackle complex reasoning remains largely unexplored. From existing work, two primary challenges emerge in this setting: (i) the dependence on an external verifier indicating a notable gap from intrinsic reasoning of human intelligence without any external feedback, and (ii) the lack of an efficient search algorithm. In this paper, we introduce the Verifier-free Test-time Scalable Diffusion Model (VFScale) to achieve scalable intrinsic reasoning, which equips number-of-sample test-time scaling with the intrinsic energy function of diffusion models as the verifier. Concretely, VFScale comprises two key innovations to address the aforementioned challenges. On the training side, VFScale consists of a novel MRNCL loss and a KL regularization to improve the energy landscape, ensuring that the learned energy function itself serves as a reliable verifier. On the inference side, VFScale integrates the denoising process with a novel hybrid Monte Carlo Tree Search (hMCTS) to improve search efficiency. On challenging reasoning tasks of Maze and Sudoku, we demonstrate the effectiveness of VFScale's training objective and scalable inference method. In particular, trained with Maze sizes of up to $6\times6$, our VFScale solves 88% of Maze problems with much larger sizes of $15\times15$, while standard diffusion models completely fail. The code can be found at https://github.com/AI4Science-WestlakeU/VFScale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。