用噪声超网络加速扩散模型推理,省时又保质。
Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models
- 用超网络动态调整初始噪声,替代耗时的测试时优化
- 在1/10计算成本下,恢复90%以上的生成质量提升
- 适合需要快速高质生成的部署场景
测试时扩展新范式在大语言模型和生成视觉模型中取得显著突破,通过推理时增加计算来应对复杂任务。然而,该方法导致计算时间大幅增加,难以实用。本文提出将测试时扩展知识融入训练后模型的解决方案:用噪声超网络取代扩散模型中的奖励引导测试时噪声优化,直接调节初始输入噪声。我们构建了一个理论严谨的框架,通过可计算的噪声空间目标学习偏置分布,使轻量化生成器在保持基础模型保真度的同时优化理想特性。实验表明,该方法以极低计算开销恢复了显式测试时优化约90%的质量增益。代码已开源。
原文摘要 · Abstract (English)
The new paradigm of test-time scaling has yielded remarkable breakthroughs in Large Language Models (LLMs) (e.g. reasoning models) and in generative vision models, allowing models to allocate additional computation during inference to effectively tackle increasingly complex problems. Despite the improvements of this approach, an important limitation emerges: the substantial increase in computation time makes the process slow and impractical for many applications. Given the success of this paradigm and its growing usage, we seek to preserve its benefits while eschewing the inference overhead. In this work we propose one solution to the critical problem of integrating test-time scaling knowledge into a model during post-training. Specifically, we replace reward guided test-time noise optimization in diffusion models with a Noise Hypernetwork that modulates initial input noise. We propose a theoretically grounded framework for learning this reward-tilted distribution for distilled generators, through a tractable noise-space objective that maintains fidelity to the base model while optimizing for desired characteristics. We show that our approach recovers a substantial portion of the quality gains from explicit test-time optimization at a fraction of the computational cost. Code is available at https://github.com/ExplainableML/HyperNoise
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。