首个将记忆令牌嵌入卷积主干的视频质量评估模型,实现动态注意力追踪。
Convolutions Need Registers Too: HVS-Inspired Dynamic Attention for Video Quality Assessment
- 在卷积网络中引入可学习记忆令牌,作为全局上下文载体。
- 生成时变显著性图,在无运动估计下跟踪视觉焦点变化。
- 支持实时推理(1080p下387.7 FPS),适合流媒体系统应用。
无参考视频质量评估(NR-VQA)在缺乏参考视频的情况下估计感知质量,通常具有挑战性。尽管近期方法利用显著性或Transformer注意力,但它们仅通过静态映射作为辅助输入处理视频信号的全局上下文,而非将上下文从根本上嵌入视频序列的特征提取中。本文提出动态注意力与全局记忆令牌结合的视频质量评估框架(DAGR-VQA),首次将记忆令牌直接整合至卷积主干,实现时空动态显著性预测。通过可学习的记忆令牌作为全局上下文载体,模型实现受人类视觉系统启发的动态注意力机制,生成随时间自适应的显著性图,无需显式运动估计即可追踪显著区域。模型将动态显著性图与RGB输入融合,通过时序Transformer分析空间数据,实现感知一致的质量评估。在LSVQ、KonVid-1k、LIVE-VQC和YouTube-UGC数据集上的全面测试表明,性能高度竞争力,优于多数顶级基线。消融实验显示,记忆令牌的引入促进了稳定且时序一致的注意力机制发展。在1080p下达到387.7 FPS的高效推理速度,满足多媒体流媒体系统等实时应用需求。
原文摘要 · Abstract (English)
No-reference video quality assessment (NR-VQA) estimates perceptual quality without a reference video, which is often challenging. While recent techniques leverage saliency or transformer attention, they merely address global context of the video signal by using static maps as auxiliary inputs rather than embedding context fundamentally within feature extraction of the video sequence. We present Dynamic Attention with Global Registers for Video Quality Assessment (DAGR-VQA), the first framework integrating register-token directly into a convolutional backbone for spatio-temporal, dynamic saliency prediction. By embedding learnable register tokens as global context carriers, our model enables dynamic, HVS-inspired attention, producing temporally adaptive saliency maps that track salient regions over time without explicit motion estimation. Our model integrates dynamic saliency maps with RGB inputs, capturing spatial data and analyzing it through a temporal transformer to deliver a perceptually consistent video quality assessment. Comprehensive tests conducted on the LSVQ, KonVid-1k, LIVE-VQC, and YouTube-UGC datasets show that the performance is highly competitive, surpassing the majority of top baselines. Research on ablation studies demonstrates that the integration of register tokens promotes the development of stable and temporally consistent attention mechanisms. Achieving an efficiency of 387.7 FPS at 1080p, DAGR-VQA demonstrates computational performance suitable for real-time applications like multimedia streaming systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。