arXiv:2412.04930cs.CVcs.LG2024-12被引 1

无需标注数据,通过分解视频图层实现去雾、重光照等编辑。

Video Decomposition Prior: A Methodology to Decompose Videos into Layers

  • 将视频分解为多层颜色与透明度图,基于输入视频自身运动和外观进行处理。
  • 在DAVIS、REVIDE等数据集上实现去雾与重光照新基准,提升视觉质量。
  • 适合需要少数据或无标注数据的视频编辑场景,如专业影视后期。

在视频增强与编辑领域,多数深度学习方法依赖大量带标注的输入-真值序列对,但在去雾、重光照等任务中,难以获取相同运动与视角的配对数据。此外,这些方法在测试分布与训练分布不一致时性能下降。本文提出视频分解先验(VDP)框架,借鉴专业视频剪辑实践,不依赖特定任务的数据收集,而是利用输入视频自身的运动与外观信息,将其分解为多个RGB图层及对应透明度。通过分别操控各图层实现目标效果。本方法应用于视频对象分割、去雾与重光照任务,并提出一种新的对数形式分解公式,显著提升重光照效果。优化该公式时,重光照特性自然涌现。我们在DAVIS、REVIDE与SDSD标准数据集上评估,且在多样互联网视频上展示定性结果。

原文摘要 · Abstract (English)

In the evolving landscape of video enhancement and editing methodologies, a majority of deep learning techniques often rely on extensive datasets of observed input and ground truth sequence pairs for optimal performance. Such reliance often falters when acquiring data becomes challenging, especially in tasks like video dehazing and relighting, where replicating identical motions and camera angles in both corrupted and ground truth sequences is complicated. Moreover, these conventional methodologies perform best when the test distribution closely mirrors the training distribution. Recognizing these challenges, this paper introduces a novel video decomposition prior `VDP' framework which derives inspiration from professional video editing practices. Our methodology does not mandate task-specific external data corpus collection, instead pivots to utilizing the motion and appearance of the input video. VDP framework decomposes a video sequence into a set of multiple RGB layers and associated opacity levels. These set of layers are then manipulated individually to obtain the desired results. We addresses tasks such as video object segmentation, dehazing, and relighting. Moreover, we introduce a novel logarithmic video decomposition formulation for video relighting tasks, setting a new benchmark over the existing methodologies. We observe the property of relighting emerge as we optimize for our novel relighting decomposition formulation. We evaluate our approach on standard video datasets like DAVIS, REVIDE, & SDSD and show qualitative results on a diverse array of internet videos. Project Page - https://www.cs.umd.edu/~gauravsh/video_decomposition/index.html for video results.

视频分解去雾重光照无监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。