arXiv:2504.16788cs.CVcs.AI2025-04被引 2

用视觉+文本融合模型,让机器看懂视频并生成自然描述。

Towards Explainable AI: Multi-Modal Transformer for Video-based Image Description Generation

  • 融合视觉与文本模态,用GPT-2生成视频描述。
  • 在BDD-X和MSVD数据集上,BLEU-4达0.755/0.778。
  • 适合需要可解释性视频理解的智能监控场景。

理解与分析视频动作对生成有洞察力且上下文相关的描述至关重要,尤其在智能监控和自主系统等视频应用中。本文提出一种新框架,通过结合文本与视觉模态,从视频数据集中生成自然语言描述。该架构利用ResNet50从Microsoft Research Video Description Corpus(MSVD)和Berkeley DeepDrive eXplanation(BDD-X)数据集的视频帧中提取视觉特征,将这些特征转化为补丁嵌入,并输入基于生成式预训练变换器-2(GPT-2)的编码器-解码器模型。为对齐文本与视觉表示并保证高质量描述生成,系统采用多头自注意力和交叉注意力机制。通过BLEU(1-4)、CIDEr、METEOR和ROUGE-L评估模型性能。所提框架在传统方法基础上显著提升:在BDD-X数据集上,BLEU-4为0.755,CIDEr为1.235,METEOR为0.312,ROUGE-L为0.782;在MSVD数据集上,对应分数分别为0.778、1.315、0.329和0.795。该研究通过生成类人、上下文相关描述,增强可解释性,推动可解释人工智能发展。

原文摘要 · Abstract (English)

Understanding and analyzing video actions are essential for producing insightful and contextualized descriptions, especially for video-based applications like intelligent monitoring and autonomous systems. The proposed work introduces a novel framework for generating natural language descriptions from video datasets by combining textual and visual modalities. The suggested architecture makes use of ResNet50 to extract visual features from video frames that are taken from the Microsoft Research Video Description Corpus (MSVD), and Berkeley DeepDrive eXplanation (BDD-X) datasets. The extracted visual characteristics are converted into patch embeddings and then run through an encoder-decoder model based on Generative Pre-trained Transformer-2 (GPT-2). In order to align textual and visual representations and guarantee high-quality description production, the system uses multi-head self-attention and cross-attention techniques. The model's efficacy is demonstrated by performance evaluation using BLEU (1-4), CIDEr, METEOR, and ROUGE-L. The suggested framework outperforms traditional methods with BLEU-4 scores of 0.755 (BDD-X) and 0.778 (MSVD), CIDEr scores of 1.235 (BDD-X) and 1.315 (MSVD), METEOR scores of 0.312 (BDD-X) and 0.329 (MSVD), and ROUGE-L scores of 0.782 (BDD-X) and 0.795 (MSVD). By producing human-like, contextually relevant descriptions, strengthening interpretability, and improving real-world applications, this research advances explainable AI.

视频描述多模态可解释AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。