通过时空采样大幅压缩视频信息,实现低延迟在线质量评估
Video Quality Assessment for Online Processing: From Spatial to Temporal Sampling
- 联合进行时空维度采样,仅保留关键帧与区域信息
- 在6个数据集上验证,丢弃大部分视频内容仍保持可接受性能
- 设计极简在线模型,适合实时视频质量检测场景
随着多媒体处理与深度学习技术的快速发展,视频质量评估(VQA)取得了显著进展。尽管研究已从高效映射模型转向多样化方向,但对时空建模中有效性和效率权衡的深入探索仍不足。鉴于视频具有高度冗余性,本文从时空联合采样的视角出发,探究在保证可接受性能前提下,输入VQA模型所需的最小信息量。为此,我们对视频在空间和时间维度上进行剧烈采样,将严重压缩后的视频输入稳定型VQA模型。在六个公开视频质量数据库上进行了全面的联合时空采样实验,结果表明:即使舍弃绝大部分视频信息,模型性能仍可接受。此外,基于该采样策略,我们首次尝试设计一种在线VQA模型,其结构仅包含最简化的空间特征提取器、时间特征融合模块和全局质量回归模块。通过定量与定性实验,验证了该简化模型在降低输入复杂度下的可行性。
原文摘要 · Abstract (English)
With the rapid development of multimedia processing and deep learning technologies, especially in the field of video understanding, video quality assessment (VQA) has achieved significant progress. Although researchers have moved from designing efficient video quality mapping models to various research directions, in-depth exploration of the effectiveness-efficiency trade-offs of spatio-temporal modeling in VQA models is still less sufficient. Considering the fact that videos have highly redundant information, this paper investigates this problem from the perspective of joint spatial and temporal sampling, aiming to seek the answer to how little information we should keep at least when feeding videos into the VQA models while with acceptable performance sacrifice. To this end, we drastically sample the video's information from both spatial and temporal dimensions, and the heavily squeezed video is then fed into a stable VQA model. Comprehensive experiments regarding joint spatial and temporal sampling are conducted on six public video quality databases, and the results demonstrate the acceptable performance of the VQA model when throwing away most of the video information. Furthermore, with the proposed joint spatial and temporal sampling strategy, we make an initial attempt to design an online VQA model, which is instantiated by as simple as possible a spatial feature extractor, a temporal feature fusion module, and a global quality regression module. Through quantitative and qualitative experiments, we verify the feasibility of online VQA model by simplifying itself and reducing input.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。