融合音频视频特征的智能质量预测模型，提升多模态内容评估精度。
Attentive AV-FusionNet: Audio-Visual Quality Prediction with Hybrid Attention
- 结合生成式语音听觉特征与人工设计视频评估特征，构建混合表征
- 通过注意力机制捕捉跨模态与模态内关系，实现上下文感知的质量判断
- 可量化各模态贡献度，适用于自适应码率分配场景
我们提出一种基于深度学习的音视频质量（AVQ）预测模型，利用当前先进单模态预测器的内部特征。不同于以往依赖简单融合策略的方法，本模型采用混合表示：将学习得到的生成式机器听觉（GML）音频特征与人工设计的视频多方法评估融合（VMAF）视频特征相结合。注意力机制捕捉跨模态交互与模态内关系，生成上下文感知的质量表征。一个模态相关性估计器可量化每种模态在不同内容下的贡献，有望支持自适应码率分配。实验表明，该模型在多种内容类型下均显著提升了音视频质量预测的准确性和鲁棒性。
原文摘要 · Abstract (English)
We introduce a novel deep learning-based audio-visual quality (AVQ) prediction model that leverages internal features from state-of-the-art unimodal predictors. Unlike prior approaches that rely on simple fusion strategies, our model employs a hybrid representation that combines learned Generative Machine Listener (GML) audio features with hand-crafted Video Multimethod Assessment Fusion (VMAF) video features. Attention mechanisms capture cross-modal interactions and intra-modal relationships, yielding context-aware quality representations. A modality relevance estimator quantifies each modality's contribution per content, potentially enabling adaptive bitrate allocation. Experiments demonstrate improved AVQ prediction accuracy and robustness across diverse content types.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。