用混合数据训练扩散模型，生成更真实连贯的全景图。
DiT360: High-Fidelity Panoramic Image Generation via Hybrid Training
- 结合透视图与全景图数据进行混合训练，提升生成质量。
- 在11项指标上优于现有方法，边界一致性与图像保真度显著提升。
- 适合需要高质量全景生成的视觉应用，如虚拟现实与摄影。
本文提出DiT360，一种基于DiT的框架，通过在透视图与全景图数据上进行混合训练，实现全景图像生成。针对生成质量中几何保真度与照片级真实感不足的问题，我们指出根源在于缺乏大规模、高质量的真实世界全景数据，这一数据驱动视角区别于以往侧重模型设计的方法。DiT360包含多个关键模块，用于域间转换与域内增强，分别作用于预VAE图像层与后VAE token层。在图像层，通过透视图引导与全景图精修融合跨域知识，提升感知质量并正则化多样性与真实感。在token层，采用多模块混合监督，包括圆形填充以保证边界连续性、偏航损失以增强旋转鲁棒性、立方体损失以感知畸变。在文本到全景、修复与外推任务上的大量实验表明，该方法在11项量化指标上均表现更优，显著提升边界一致性与图像保真度。代码已开源：https://github.com/Insta360-Research-Team/DiT360。
原文摘要 · Abstract (English)
In this work, we propose DiT360, a DiT-based framework that performs hybrid training on perspective and panoramic data for panoramic image generation. For the issues of maintaining geometric fidelity and photorealism in generation quality, we attribute the main reason to the lack of large-scale, high-quality, real-world panoramic data, where such a data-centric view differs from prior methods that focus on model design. Basically, DiT360 has several key modules for inter-domain transformation and intra-domain augmentation, applied at both the pre-VAE image level and the post-VAE token level. At the image level, we incorporate cross-domain knowledge through perspective image guidance and panoramic refinement, which enhance perceptual quality while regularizing diversity and photorealism. At the token level, hybrid supervision is applied across multiple modules, which include circular padding for boundary continuity, yaw loss for rotational robustness, and cube loss for distortion awareness. Extensive experiments on text-to-panorama, inpainting, and outpainting tasks demonstrate that our method achieves better boundary consistency and image fidelity across eleven quantitative metrics. Our code is available at https://github.com/Insta360-Research-Team/DiT360.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。