提出新型奖励机制,让生成图像时的中间步骤也能准确评估质量。
FR-TTS: Test-Time Scaling for NTP-based Image Generation with Effective Filling-based Reward Signal
- 用填充方案补全中间生成序列,估算其未来质量
- 实验显示该方法在多个模型上超越现有基准
- 适合需要高质量图像生成的科研与应用开发者
测试时缩放(TTS)已成为图像生成中的主流技术,通过增加并行样本数量并利用预训练奖励模型筛选,显著提升输出质量。然而,将其应用于下一词预测(NTP)范式仍面临挑战。主要障碍在于:从中间令牌序列解码出的图像奖励与完整生成图像的奖励相关性低,导致这些不完整的中间表示难以有效指导剪枝方向。为此,本文提出基于填充的奖励(FR),通过寻找并应用合理的填充方案来补全序列,从而估计中间样本的近似未来轨迹。实验证明,该方法在中间样本与最终样本奖励间的相关系数,以及令牌置信度等内在信号上均表现优异,能可靠评估中间样本质量。在此基础上,提出FR-TTS,一种高效的缩放策略,通过搜索优质填充方案,并引入动态加权的多样性奖励,实现对中间样本的全面评估。在多个基准和奖励模型上,FR-TTS均表现出优越性能。代码已开源。
原文摘要 · Abstract (English)
Test-time scaling (TTS) has become a prevalent technique in image generation, significantly boosting output quality by expanding the number of parallel samples and filtering them using pre-trained reward models. However, applying this powerful methodology to the next-token prediction (NTP) paradigm remains challenging. The primary obstacle is the low correlation between the reward of an image decoded from an intermediate token sequence and the reward of the fully generated image. Consequently, these incomplete intermediate representations prove to be poor indicators for guiding the pruning direction, a limitation that stems from their inherent incompleteness in scale or semantic content. To effectively address this critical issue, we introduce the Filling-Based Reward (FR). This novel design estimates the approximate future trajectory of an intermediate sample by finding and applying a reasonable filling scheme to complete the sequence. Both the correlation coefficient between rewards of intermediate samples and final samples, as well as multiple intrinsic signals like token confidence, indicate that the FR provides an excellent and reliable metric for accurately evaluating the quality of intermediate samples. Building upon this foundation, we propose FR-TTS, a sophisticated scaling strategy. FR-TTS efficiently searches for good filling schemes and incorporates a diversity reward with a dynamic weighting schedule to achieve a balanced and comprehensive evaluation of intermediate samples. We experimentally validate the superiority of FR-TTS over multiple established benchmarks and various reward models. Code is available at \href{https://github.com/xuhang07/FR-TTS}{https://github.com/xuhang07/FR-TTS}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。