用2D高斯点云实现任意尺度超分辨率,兼顾质量与效率。
Generalized and Efficient 2D Gaussian Splatting for Arbitrary-scale Super-Resolution
- 设计前馈架构,根据低分辨率图像生成条件化高斯点。
- 在不同缩放因子下实现端到端训练,支持任意尺度超分。
- 基于GPU的可微分渲染,比传统方法快3倍以上。
隐式神经表示(INR)已成功应用于任意尺度超分辨率(ASR),但其需多次查询多层感知机并逐像素渲染,导致表达能力弱且计算效率低。最近,高斯点阵(GS)在3D任务中展现出优于INR的视觉质量和渲染速度,启发我们探索其在ASR中的应用。然而,直接使用原版GS存在挑战:其为基于优化的过拟合方法,而ASR要求模型泛化至不同图像和缩放因子。为此,我们提出两项创新:首先,设计前馈架构,以生成输入低分辨率图像对应的图像条件化高斯点,每个高斯点可拟合复杂纹理的形状与方向,具备强大表达能力;其次,实现高效的可微分2D GPU/CUDA级尺度感知光栅化,通过从预测的连续高斯点采样离散RGB值来渲染超分辨图像。经端到端训练,所提出的GSASR模型可对任意图像和未见缩放因子执行ASR。大量实验验证了方法有效性。代码与模型已开源。
原文摘要 · Abstract (English)
Implicit Neural Representations (INR) have been successfully employed for Arbitrary-scale Super-Resolution (ASR). However, INR-based models need to query the multi-layer perceptron module numerous times and render a pixel in each query, resulting in insufficient representation capability and low computational efficiency. Recently, Gaussian Splatting (GS) has shown its advantages over INR in both visual quality and rendering speed in 3D tasks, which motivates us to explore whether GS can be employed for the ASR task. However, directly applying GS to ASR is exceptionally challenging because the original GS is an optimization-based method through overfitting each single scene, while in ASR we aim to learn a single model that can generalize to different images and scaling factors. We overcome these challenges by developing two novel techniques. Firstly, to generalize GS for ASR, we elaborately design an architecture to predict the corresponding image-conditioned Gaussians of the input low-resolution image in a feed-forward manner. Each Gaussian can fit the shape and direction of an area of complex textures, showing powerful representation capability. Secondly, we implement an efficient differentiable 2D GPU/CUDA-based scale-aware rasterization to render super-resolved images by sampling discrete RGB values from the predicted continuous Gaussians. Via end-to-end training, our optimized network, namely GSASR, can perform ASR for any image and unseen scaling factors. Extensive experiments validate the effectiveness of our proposed method. The code and models are available at https://github.com/ChrisDud0257/GSASR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。