arXiv:2410.11506cs.CV2024-10AAAI被引 4

解决全景视频超分辨率中的时空失真问题,提升画质与流畅度。

Spatio-Temporal Distortion Aware Omnidirectional Video Super-Resolution

  • 设计时空联合对齐网络,缓解投影畸变和帧间闪烁。
  • 在新构建的全景视频数据集上实现更优的视觉保真度与动态平滑性。
  • 适合虚拟现实、元宇宙等需要高质量全景内容的场景使用。

全景视频(ODVs)通过捕捉360°场景提供沉浸式视觉体验。随着虚拟/增强现实、元宇宙及生成式AI的快速发展,对高质量全景视频的需求日益增长。然而,由于视场宽广以及采集设备与传输带宽限制,全景视频常存在低分辨率问题。尽管视频超分辨率(SR)是有效的画质增强技术,但现有方法在应用于全景视频时受限于性能上限与实际泛化能力。为缓解全景视频的空间投影畸变与时间闪烁问题,本文提出时空失真感知网络(STDAN),采用时空联合对齐与重建框架。具体地,引入时空连续对齐(STCA)以减轻离散几何伪影,并行实现时间对齐;进一步提出交错多帧重建(IMFR)以增强时间一致性;同时采用纬度显著性自适应(LSA)权重,聚焦纹理复杂且人眼关注区域。通过探索时空联合框架与真实观看策略,STDAN在新构建的ODV-SR数据集上有效提升时空一致性,且计算成本可控。大量实验表明,其在提升全景视频视觉保真度与动态平滑性方面优于现有最优方法。

原文摘要 · Abstract (English)

Omnidirectional videos (ODVs) provide an immersive visual experience by capturing the 360° scene. With the rapid advancements in virtual/augmented reality, metaverse, and generative artificial intelligence, the demand for high-quality ODVs is surging. However, ODVs often suffer from low resolution due to their wide field of view and limitations in capturing devices and transmission bandwidth. Although video super-resolution (SR) is a capable video quality enhancement technique, the performance ceiling and practical generalization of existing methods are limited when applied to ODVs due to their unique attributes. To alleviate spatial projection distortions and temporal flickering of ODVs, we propose a Spatio-Temporal Distortion Aware Network (STDAN) with joint spatio-temporal alignment and reconstruction. Specifically, we incorporate a spatio-temporal continuous alignment (STCA) to mitigate discrete geometric artifacts in parallel with temporal alignment. Subsequently, we introduce an interlaced multi-frame reconstruction (IMFR) to enhance temporal consistency. Furthermore, we employ latitude-saliency adaptive (LSA) weights to focus on regions with higher texture complexity and human-watching interest. By exploring a spatio-temporal jointly framework and real-world viewing strategies, STDAN effectively reinforces spatio-temporal coherence on a novel ODV-SR dataset and ensures affordable computational costs. Extensive experimental results demonstrate that STDAN outperforms state-of-the-art methods in improving visual fidelity and dynamic smoothness of ODVs.

全景视频超分辨率时空对齐图像重建

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。