arXiv:2505.03261cs.CVeess.IV2025-05被引 1

用扩散模型提取特征,提升视频质量评估的准确性与泛化能力

DiffVQA: Video Quality Assessment Using Diffusion Feature Extractor

  • 用扩散模型作为特征提取器,通过控制模块重建输入帧
  • 在多个数据集上表现优于CNN和ViT,跨数据集泛化能力更强
  • 结合Mamba模块捕捉长期时序信息,适合真实场景下的视频评估

视频质量评估(VQA)旨在基于感知失真和人类偏好评估视频质量。尽管现有基于卷积神经网络(CNN)和视觉变换器(ViT)的方法表现良好,但在多样化的现实场景中仍难以与人类感知保持一致,这主要受限于现有数据集规模和多样性不足。为此,本文提出一种新框架DiffVQA,利用在大规模数据上预训练的扩散模型所具备的强大泛化能力。该框架通过控制模块使扩散模型重建相同的输入帧,并将其用于提取语义特征(通过缩放分支)和失真特征(通过裁剪分支)。为增强对长时序动态的建模能力,引入并行的Mamba模块,提取时序一致性增强特征,并与扩散特征融合以预测最终评分。在多个数据集上的实验表明,DiffVQA在同数据集评估中表现优异,且跨数据集泛化能力突出。结果证实,将扩散模型作为特征提取器可显著提升VQA性能,优于传统的CNN和ViT骨干网络。

原文摘要 · Abstract (English)

Video Quality Assessment (VQA) aims to evaluate video quality based on perceptual distortions and human preferences. Despite the promising performance of existing methods using Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs), they often struggle to align closely with human perceptions, particularly in diverse real-world scenarios. This challenge is exacerbated by the limited scale and diversity of available datasets. To address this limitation, we introduce a novel VQA framework, DiffVQA, which harnesses the robust generalization capabilities of diffusion models pre-trained on extensive datasets. Our framework adapts these models to reconstruct identical input frames through a control module. The adapted diffusion model is then used to extract semantic and distortion features from a resizing branch and a cropping branch, respectively. To enhance the model's ability to handle long-term temporal dynamics, a parallel Mamba module is introduced, which extracts temporal coherence augmented features that are merged with the diffusion features to predict the final score. Experiments across multiple datasets demonstrate DiffVQA's superior performance on intra-dataset evaluations and its exceptional generalization across datasets. These results confirm that leveraging a diffusion model as a feature extractor can offer enhanced VQA performance compared to CNN and ViT backbones.

视频质量评估扩散模型Mamba特征提取

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。