用并行退火法改进生成模型的初始采样,让高奖励区域更容易被找到。
Parallel Tempering Initial Sampling in Inference-Time Reward Alignment

- 通过多链并行退火,跨不同奖励温度同步采样以提升探索能力。
- 在布局到图像和数量感知生成任务中,复杂提示下的对齐质量显著提升。
- 特别适合需要精准满足复杂用户需求的生成场景,如艺术设计、工业建模。
推理时奖励对齐可使预训练的扩散模型和基于流的生成模型在不重新训练的情况下满足用户指定的奖励。近期,序列蒙特卡洛(SMC)框架因其迭代过滤与传播多个粒子而成为该任务的强大工具。然而,我们发现标准SMC方法性能不佳,原因是粒子从标准先验初始化,而在复杂奖励景观中高奖励区域极为稀少。此外,即使最新的奖励感知初始采样方法仍易陷入局部模式,因为复杂奖励景观通常具有多模态特性。为克服这些限制,我们提出PATHS(PArallel Tempering for High-complexity reward Sampling),一种通过并行退火耦合多个采样链的新初始方法。PATHS维护一系列奖励退火链,并周期性执行梅特罗波利斯交换,从而在平坦化的奖励景观中实现高效探索,有效缓解模式陷阱问题。分析表明,该机制显著增强了有限预算下对罕见高奖励区域的探索能力。在布局到图像和数量感知生成任务上的实验显示,PATHS在复杂提示下实现了对齐质量的持续提升。
原文摘要 · Abstract (English)
Inference-time reward alignment steers pretrained diffusion and flow-based generative models to satisfy user-specified rewards without retraining. Recently, Sequential Monte Carlo (SMC) has emerged as a powerful framework for this task by iteratively filtering and propagating multiple particles. However, we show that standard SMC-based methods often suffer from poor performance because they initialize particles from a standard prior, whereas high-reward regions in complex reward landscapes are extremely rare. Further, we show that even recent reward-aware initial sampling approaches remain vulnerable to getting trapped in local modes, as complex reward landscapes are often multi-modal. To overcome these limitations, we propose PATHS (PArallel Tempering for High-complexity reward Sampling), a novel initialization method that couples multiple sampling chains through parallel tempering. PATHS maintains a ladder of reward-tempered chains and periodically performs Metropolis swaps, enabling efficient exploration across flattened reward landscapes, thereby mitigating the mode-trapping issues. Our analysis reveals that this mechanism substantially enhances the finite-budget exploration of rare, high-reward regions that are typically challenging to sample. Experiments on layout-to-image and quantity-aware generation show that PATHS achieves consistent gains in alignment quality, particularly on complex prompts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。