通过优化无文本嵌入实现图像生成模型的测试时对齐,避免奖励作弊。
Test-Time Alignment of Text-to-Image Diffusion Models via Null-Text Embedding Optimisation
- 在文本嵌入空间优化无条件嵌入,实现语义一致的对齐
- 相比现有方法,既不欠优化也不过优化,保持生成质量
- 适合需要精准控制生成内容但无法重训练的场景
测试时对齐(TTA)旨在推理阶段适应特定奖励函数。然而,现有方法往往出现优化不足或过度优化(奖励作弊)的问题。本文提出无文本测试时对齐(Null-TTA),通过优化分类器无关引导中的无条件嵌入,而非操纵潜在变量或噪声。由于文本嵌入空间具有结构化的语义特性,该方法确保对齐发生在语义一致的流形上,防止利用非语义噪声模式提升奖励的奖励作弊行为。无条件嵌入作为生成分布的锚点,使Null-TTA直接引导模型生成分布向目标奖励靠近,无需更新模型参数。得益于这些优势,Null-TTA在目标测试时对齐任务中达到当前最优性能,同时保持良好的跨奖励泛化能力。这确立了语义空间优化作为TTA的一种有效且原则性的新范式。
原文摘要 · Abstract (English)
Test-time alignment (TTA) aims to adapt models to specific rewards during inference. However, existing methods tend to either under-optimise or over-optimise (reward hack) the target reward function. We propose Null-Text Test-Time Alignment (Null-TTA), which aligns diffusion models by optimising the unconditional embedding in classifier-free guidance, rather than manipulating latent or noise variables. Due to the structured semantic nature of the text embedding space, this ensures alignment occurs on a semantically coherent manifold and prevents reward hacking (exploiting non-semantic noise patterns to improve the reward). Since the unconditional embedding in classifier-free guidance serves as the anchor for the model's generative distribution, Null-TTA directly steers model's generative distribution towards the target reward rather than just adjusting the samples, even without updating model parameters. Thanks to these desirable properties, we show that Null-TTA achieves state-of-the-art target test-time alignment while maintaining strong cross-reward generalisation. This establishes semantic-space optimisation as an effective and principled novel paradigm for TTA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。