将图像级人脸修复扩展到视频,解决动态抖动与闪烁问题。
Analysis and Benchmarking of Extending Blind Face Image Restoration to Videos
- 构建真实低质人脸视频基准数据集RFV-LQ,评估现有算法表现
- 发现视频修复中存在面部部件剧烈抖动和帧间噪声形状闪烁
- 提出时序一致性网络TCN,可无缝接入主流修复模型
近年来,盲人脸修复在静态图像上已取得高质量恢复成果。然而,将其拓展至视频场景的研究仍较少,主要受限于缺乏全面且公平的评测基准。本文首次构建了一个真实世界低质量人脸视频基准(RFV-LQ),评估了多个先进的基于图像的人脸修复算法,并对将盲人脸图像修复方法扩展至退化人脸视频所面临的收益与挑战进行了系统性分析。分析揭示了两个关键问题:面部组件显著抖动以及帧间噪声-形状闪烁。为解决这些问题,我们提出一种时序一致性网络(TCN)结合对齐平滑策略,有效降低修复视频中的抖动与闪烁。该模块具有高度灵活性,可无缝集成至当前最先进的图像修复算法中,最大限度保持原有图像修复质量。大量实验验证了所提TCN及对齐平滑操作的有效性与效率。
原文摘要 · Abstract (English)
Recent progress in blind face restoration has resulted in producing high-quality restored results for static images. However, efforts to extend these advancements to video scenarios have been minimal, partly because of the absence of benchmarks that allow for a comprehensive and fair comparison. In this work, we first present a fair evaluation benchmark, in which we first introduce a Real-world Low-Quality Face Video benchmark (RFV-LQ), evaluate several leading image-based face restoration algorithms, and conduct a thorough systematical analysis of the benefits and challenges associated with extending blind face image restoration algorithms to degraded face videos. Our analysis identifies several key issues, primarily categorized into two aspects: significant jitters in facial components and noise-shape flickering between frames. To address these issues, we propose a Temporal Consistency Network (TCN) cooperated with alignment smoothing to reduce jitters and flickers in restored videos. TCN is a flexible component that can be seamlessly plugged into the most advanced face image restoration algorithms, ensuring the quality of image-based restoration is maintained as closely as possible. Extensive experiments have been conducted to evaluate the effectiveness and efficiency of our proposed TCN and alignment smoothing operation. Project page: https://wzhouxiff.github.io/projects/FIR2FVR/FIR2FVR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。