轻量级GAN实时提升游戏画面真实感，兼顾速度与保真。
HyPER-GAN: Hybrid Patch-Based Image-to-Image Translation for Real-Time Photorealism Enhancement in Game Engines
- 用轻量U-Net生成器+混合训练策略，实现低延迟推理。
- 1080p下帧率提升6倍，真实感增强且语义一致。
- 适合游戏引擎实时渲染、仿真数据生成场景。
生成模型被广泛用于提升游戏引擎渲染图像的真实感，以支持视觉合成数据生成与仿真应用。然而，现有方法常引入影响原始场景内容的伪影，并需高算力，限制其在训练/评估数据增强及渲染管线中的应用。本文提出HyPER-GAN：一种基于轻量级U-Net生成器的混合块图像到图像翻译框架，支持实时推理。该框架采用成对渲染图与真实感增强图进行训练，并引入新型混合训练策略，融合未配对的真实世界图像匹配块，以提升内容保真度并进一步增强视觉真实感。实验表明，相比当前最先进轻量级成对图像翻译方法，HyPER-GAN在1080p下帧率提升6倍，同时在同引擎与跨引擎评估中均显著提升渲染图像真实感，且不明显损害语义一致性。此外，该方法保持时间一致性，混合训练策略在同引擎下提升内容保真度与视觉真实感，在跨引擎评估中增强鲁棒性。代码与预训练模型已公开：https://github.com/stefanos50/HyPER-GAN
原文摘要 · Abstract (English)
Generative models are increasingly used in video game engines to enhance the photorealism of rendered images for visual synthetic data generation and simulation applications. However, they often introduce artifacts that alter the content of the original rendered scenes and require high computational resources, which limit their utilization for the photorealism enhancement of training and evaluation data, as well as their integration in the rendering pipelines of game engines. In this paper, we propose Hybrid Patch Enhanced Realism Generative Adversarial Network (HyPER-GAN), a hybrid image-to-image translation framework that is based on a lightweight U-Net-style generator capable of performing real-time inference. The framework is trained using paired rendered and photorealism-enhanced images, complemented by a novel hybrid training strategy that incorporates matched patches from unpaired real-world images to improve content preservation and further enhance the visual realism that can be achieved by the lightweight generator. Experimental results demonstrate that HyPER-GAN achieves a 6x increase in frames per second at 1080p in comparison with state-of-the-art lightweight paired image-to-image translation methods, while also increasing, in both within- and cross-engine evaluations, the photorealism of the rendered images without significantly compromising semantic consistency. Moreover, it is illustrated that HyPER-GAN maintains temporal consistency and that the proposed hybrid training strategy improves content preservation and visual realism in within-engine and increases the robustness in cross-engine evaluations compared to training the framework solely with paired rendered and photorealism-enhanced images. Code and pretrained models are publicly available at: https://github.com/stefanos50/HyPER-GAN
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。