让图像生成自动匹配复杂提示,算得少还更准。
RAISE: Requirement-Adaptive Evolutionary Refinement for Training-Free Text-to-Image Alignment
- 生成时动态调整计算量,按提示难易分配资源
- 比现有方法少30%-40%生成样本,减少80%视觉语言模型调用
- 无需训练,适配多种模型,适合复杂提示生成任务
当前文本到图像扩散模型虽具高度真实感,但复杂提示(含多对象、关系与细粒度属性)的对齐仍存挑战。现有无训练推理增强方法依赖固定迭代次数,无法适应提示难度;而基于反思的微调模型需精心构建反思数据集,并对扩散与视觉语言模型联合微调,常在特定路径上过拟合,缺乏跨模型泛化能力。本文提出RAISE(需求自适应自我提升演化框架),一种无训练、需求驱动的演化生成方法。将图像生成建模为需求驱动的自适应扩展过程,推理时通过提示重写、噪声重采样、指令编辑等多样化修正动作演化候选集,每代均依据结构化需求清单验证,动态识别未满足项并仅在必要处追加计算。该方法实现计算与语义查询复杂度的自适应匹配。在GenEval和DrawBench测试中,RAISE达到0.94的总体对齐分数,优于现有缩放与反思调优基线,生成样本减少30%-40%,视觉语言模型调用减少80%,展现高效、通用且模型无关的多轮自优化能力。代码已公开于https://github.com/LiyaoJiang1998/RAISE。
原文摘要 · Abstract (English)
Recent text-to-image (T2I) diffusion models achieve remarkable realism, yet faithful prompt-image alignment remains challenging, particularly for complex prompts with multiple objects, relations, and fine-grained attributes. Existing training-free inference-time scaling methods rely on fixed iteration budgets that cannot adapt to prompt difficulty, while reflection-tuned models require carefully curated reflection datasets and extensive joint fine-tuning of diffusion and vision-language models, often overfitting to reflection paths data and lacking transferability across models. We introduce RAISE (Requirement-Adaptive Self-Improving Evolution), a training-free, requirement-driven evolutionary framework for adaptive T2I generation. RAISE formulates image generation as a requirement-driven adaptive scaling process, evolving a population of candidates at inference time through a diverse set of refinement actions-including prompt rewriting, noise resampling, and instructional editing. Each generation is verified against a structured checklist of requirements, enabling the system to dynamically identify unsatisfied items and allocate further computation only where needed. This achieves adaptive test-time scaling that aligns computational effort with semantic query complexity. On GenEval and DrawBench, RAISE attains state-of-the-art alignment (0.94 overall GenEval) while incurring fewer generated samples (reduced by 30-40%) and VLM calls (reduced by 80%) than prior scaling and reflection-tuned baselines, demonstrating efficient, generalizable, and model-agnostic multi-round self-improvement. Code is available at https://github.com/LiyaoJiang1998/RAISE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。