针对全景视频分辨率低问题,提出新型深度学习模型提升画质。
Omnidirectional Video Super-Resolution using Deep Learning
- 用球面注意力机制建模,避开传统对齐方式
- 在自建360VDS数据集上超越主流超分模型
- 适合虚拟现实、全景视频处理领域研究者
全景视频(360°视频)广泛应用于虚拟现实,以提供沉浸式观看体验。然而,其空间分辨率有限,导致每个视角像素不足,影响视觉质量。传统深度学习视频超分辨率技术虽具潜力,但未解决360°视频等距投影带来的畸变问题,且高质量360°视频数据集稀缺。为此,本文构建了新的360°视频数据集360VDS,研究常规超分模型在360°视频上的可扩展性,并提出名为Spherical Signal Super-resolution with Proportioned Optimisation(S3PO)的新模型。S3PO采用循环建模与注意力机制,不依赖传统对齐操作;结合专用特征提取器与针对球面畸变设计的损失函数,在360°视频数据集上优于多数先进常规及专用超分模型。通过逐步消融实验,验证了各模块、训练策略与优化方法的有效性。
原文摘要 · Abstract (English)
Omnidirectional Videos (or 360° videos) are widely used in Virtual Reality (VR) to facilitate immersive and interactive viewing experiences. However, the limited spatial resolution in 360° videos does not allow for each degree of view to be represented with adequate pixels, limiting the visual quality offered in the immersive experience. Deep learning Video Super-Resolution (VSR) techniques used for conventional videos could provide a promising software-based solution; however, these techniques do not tackle the distortion present in equirectangular projections of 360° video signals. An additional obstacle is the limited availability of 360° video datasets for study. To address these issues, this paper creates a novel 360° Video Dataset (360VDS) with a study of the extensibility of conventional VSR models to 360° videos. This paper further proposes a novel deep learning model for 360° Video Super-Resolution (360° VSR), called Spherical Signal Super-resolution with a Proportioned Optimisation (S3PO). S3PO adopts recurrent modelling with an attention mechanism, unbound from conventional VSR techniques like alignment. With a purpose-built feature extractor and a novel loss function addressing spherical distortion, S3PO outperforms most state-of-the-art conventional VSR models and 360°~specific super-resolution models on 360° video datasets. A step-wise ablation study is presented to understand and demonstrate the impact of the chosen architectural sub-components, targeted training and optimisation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。