将密集文本生成与全景图模型结合，实现无畸变的球面图像合成。
Spherical Dense Text-to-Image Synthesis
- 通过融合多扩散技术改进全景生成模型，实现球面布局控制。
- 新数据集DSynView验证了方法在图像质量与布局匹配上的优势。
- 适合需要高质量全景图像生成的研究者和开发者使用。
近期文本到图像（T2I）生成技术虽有进展，但在布局控制与生成全向全景图像方面仍面临挑战。密集型文本到图像（DT2I）与球面文本到图像（ST2I）模型分别解决部分问题，但尚无统一方案。简单地用DT2I模型生成全景图无法产生正确的球面畸变与边界无缝衔接。本文提出无需训练的球面密集文本到图像（SDT2I）方法，通过将无训练的DT2I方法集成到微调后的全景模型中实现。具体提出MultiStitchDiffusion（MSTD）和MultiPanFusion（MPF），分别整合MultiDiffusion至StitchDiffusion和PanFusion。由于缺乏SDT2I基准，我们构建了包含球面布局的新合成数据集Dense-Synthetic-View（DSynView）以评估模型。结果表明，MSTD在图像质量、提示遵循性与布局一致性上均优于MPF；MPF生成图像多样性更高，但难以完美合成前景物体。为此，我们提出自举耦合策略并关闭前景的等距柱状投影注意力机制以改进MPF。
原文摘要 · Abstract (English)
Recent advancements in text-to-image (T2I) have improved synthesis results, but challenges remain in layout control and generating omnidirectional panoramic images. Dense T2I (DT2I) and spherical T2I (ST2I) models address these issues, but so far no unified approach exists. Trivial approaches, like prompting a DT2I model to generate panoramas can not generate proper spherical distortions and seamless transitions at the borders. Our work shows that spherical dense text-to-image (SDT2I) can be achieved by integrating training-free DT2I approaches into finetuned panorama models. Specifically, we propose MultiStitchDiffusion (MSTD) and MultiPanFusion (MPF) by integrating MultiDiffusion into StitchDiffusion and PanFusion, respectively. Since no benchmark for SDT2I exists, we further construct Dense-Synthetic-View (DSynView), a new synthetic dataset containing spherical layouts to evaluate our models. Our results show that MSTD outperforms MPF across image quality as well as prompt- and layout adherence. MultiPanFusion generates more diverse images but struggles to synthesize flawless foreground objects. We propose bootstrap-coupling and turning off equirectangular perspective-projection attention in the foreground as an improvement of MPF. Link to code https://github.com/sdt2i/spherical-dense-text-to-image
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。