arXiv:2411.16468cs.CV2024-11CVPR被引 7

提出高效视频人脸增强方法,解决模糊与闪烁问题。

Efficient Video Face Enhancement with Enhanced Spatial-Temporal Consistency

  • 基于3D-VQGAN与时空码本,实现端到端人脸视频增强
  • 在VFHQ-Test上比现有方法快2.3倍,且无闪烁现象
  • 适合实时直播、短视频修复等对速度与画质要求高的场景

人脸视频广泛存在于电影、访谈节目和直播中。由于高通信成本和带宽限制,真实在线视频常因高压缩比出现模糊和量化噪声,而人眼对人脸细节极为敏感,此类退化影响显著。尽管视频人脸增强技术进展迅速,现有方法仍面临处理时间长和时空视觉不一致(如闪烁)的问题。本文提出一种新型高效盲视频人脸增强方法,可从压缩低质量视频恢复高质量视频,并具备有效去闪烁机制。该方法基于3D-VQGAN主干网络,结合记录高质量人脸特征的时空码本及基于残差的时序信息。设计两阶段学习框架:第一阶段通过正则化缓解码本坍塌;第二阶段引入两个Transformer分别查询码本并更新低质量视频编码器。在VFHQ-Test数据集上的实验表明,该方法在效率与效果上均优于当前最先进的盲人脸视频恢复与去闪烁方法。代码已开源。

原文摘要 · Abstract (English)

As a very common type of video, face videos often appear in movies, talk shows, live broadcasts, and other scenes. Real-world online videos are often plagued by degradations such as blurring and quantization noise, due to the high compression ratio caused by high communication costs and limited transmission bandwidth. These degradations have a particularly serious impact on face videos because the human visual system is highly sensitive to facial details. Despite the significant advancement in video face enhancement, current methods still suffer from $i)$ long processing time and $ii)$ inconsistent spatial-temporal visual effects (e.g., flickering). This study proposes a novel and efficient blind video face enhancement method to overcome the above two challenges, restoring high-quality videos from their compressed low-quality versions with an effective de-flickering mechanism. In particular, the proposed method develops upon a 3D-VQGAN backbone associated with spatial-temporal codebooks recording high-quality portrait features and residual-based temporal information. We develop a two-stage learning framework for the model. In Stage \Rmnum{1}, we learn the model with a regularizer mitigating the codebook collapse problem. In Stage \Rmnum{2}, we learn two transformers to lookup code from the codebooks and further update the encoder of low-quality videos. Experiments conducted on the VFHQ-Test dataset demonstrate that our method surpasses the current state-of-the-art blind face video restoration and de-flickering methods on both efficiency and effectiveness. Code is available at \url{https://github.com/Dixin-Lab/BFVR-STC}.

视频增强人脸修复去闪烁高效模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。