无需微调,通过分层提示提升4K图像生成质量。
HiPrompt: Tuning-free Higher-Resolution Generation with Hierarchical MLLM Prompts
- 用分层提示提供全局与局部双重引导,解决高分辨率生成难题。
- 在去噪过程中分解噪声为高低频成分,实现多级提示协同控制。
- 显著减少物体重复,提升结构与纹理一致性,适合高清图像生成场景。
使用预训练扩散模型进行更高分辨率图像生成潜力巨大,但这些模型在扩展至4K及以上分辨率时,常出现物体重复和结构伪影问题。我们发现,单一提示难以有效指导多尺度生成。为此,提出无需微调的HiPrompt方法,引入分层提示机制。全局提示由用户输入提供,描述整体内容;局部提示则利用多模态大模型(MLLM)生成的局部区域描述,精细引导各区域的结构与纹理。在逆向去噪过程中,生成噪声被分解为低频与高频空间成分,并分别受多层次提示(包括详细局部描述和整体图像提示)条件约束,实现基于分层语义引导的去噪。该机制使生成过程更聚焦于局部空间区域,确保图像在高分辨率下保持一致的局部与全局语义、结构与纹理。大量实验表明,HiPrompt在高分辨率图像生成上优于现有最优方法,显著降低物体重复率并提升结构质量。
原文摘要 · Abstract (English)
The potential for higher-resolution image generation using pretrained diffusion models is immense, yet these models often struggle with issues of object repetition and structural artifacts especially when scaling to 4K resolution and higher. We figure out that the problem is caused by that, a single prompt for the generation of multiple scales provides insufficient efficacy. In response, we propose HiPrompt, a new tuning-free solution that tackles the above problems by introducing hierarchical prompts. The hierarchical prompts offer both global and local guidance. Specifically, the global guidance comes from the user input that describes the overall content, while the local guidance utilizes patch-wise descriptions from MLLMs to elaborately guide the regional structure and texture generation. Furthermore, during the inverse denoising process, the generated noise is decomposed into low- and high-frequency spatial components. These components are conditioned on multiple prompt levels, including detailed patch-wise descriptions and broader image-level prompts, facilitating prompt-guided denoising under hierarchical semantic guidance. It further allows the generation to focus more on local spatial regions and ensures the generated images maintain coherent local and global semantics, structures, and textures with high definition. Extensive experiments demonstrate that HiPrompt outperforms state-of-the-art works in higher-resolution image generation, significantly reducing object repetition and enhancing structural quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。