用混合模型缩小游戏引擎图像与真实世界差距
A Hybrid Approach for Closing the Sim2real Appearance Gap in Game Engine Synthetic Datasets
- 结合扩散模型与图像翻译技术提升合成图像真实感
- 混合方法在视觉真实感上优于单一模型，保持语义一致
- 适合需要高保真合成数据的计算机视觉研究者
视频游戏引擎已成为生成大规模视觉合成数据集的重要来源，用于训练和评估将部署于真实世界的计算机视觉算法。尽管现代游戏引擎借助光线追踪等技术显著提升了视觉保真度，合成图像与真实世界图像之间仍存在显著的sim2real外观差距，限制了合成数据在真实场景中的应用。本文研究了先进图像生成与编辑扩散模型FLUX.2-4B Klein在增强合成数据摄影真实感方面的效果，并与传统图像到图像翻译模型REGEN进行对比。进一步提出一种混合方法，融合扩散模型在几何与材质变换上的优势，以及图像翻译技术在分布匹配上的能力。实验表明，REGEN在性能上优于FLUX.2-4B Klein；而将两者结合使用，可实现比单独使用任一模型更佳的视觉真实感，同时保持语义一致性。代码已公开于https://github.com/stefanos50/Hybrid-Sim2Real。
原文摘要 · Abstract (English)
Video game engines have been an important source for generating large volumes of visual synthetic datasets for training and evaluating computer vision algorithms that are to be deployed in the real world. While the visual fidelity of modern game engines has been significantly improved with technologies such as ray-tracing, a notable sim2real appearance gap between the synthetic and the real-world images still remains, which limits the utilization of synthetic datasets in real-world applications. In this letter, we investigate the ability of a state-of-the-art image generation and editing diffusion model (FLUX.2-4B Klein) to enhance the photorealism of synthetic datasets and compare its performance against a traditional image-to-image translation model (REGEN). Furthermore, we propose a hybrid approach that combines the strong geometry and material transformations of diffusion-based methods with the distribution-matching capabilities of image-to-image translation techniques. Through experiments, it is demonstrated that REGEN outperforms FLUX.2-4B Klein and that by combining both FLUX.2-4B Klein and REGEN models, better visual realism can be achieved compared to using each model individually, while maintaining semantic consistency. The code is available at: https://github.com/stefanos50/Hybrid-Sim2Real
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。