用大模型理解视频语义,生成更连贯的摘要。
Video Summarization with Large Language Models
- 用多模态大模型将帧转为描述,再用大模型评估重要性。
- 在标准数据集上优于现有方法,关键帧选择更符合人类判断。
- 适合想用大模型做视频理解与摘要的研究者和开发者。
视频内容的爆炸式增长给高效导航、搜索和检索带来了巨大挑战,亟需先进的视频摘要技术。现有方法主要依赖视觉特征和时序动态,难以捕捉视频语义,导致摘要不完整或不连贯。为此,我们提出一种基于大语言模型(LLM)的新框架——LLMVS,利用大模型从海量数据中学习到的知识,使其能以更贴近多样语义和人类判断的方式评估视频帧的重要性,有效应对关键帧定义中的主观性问题。该方法首先通过多模态大语言模型(M-LLM)将视频帧转换为文本描述序列,再基于局部上下文中的描述,使用大模型评估每帧的重要性;随后通过全局注意力机制对全视频描述上下文中的重要性得分进行精炼,确保摘要既能反映细节,又能体现整体叙事。实验结果表明,该方法在标准基准测试中显著优于现有方法,展现了大模型在多媒体处理中的巨大潜力。
原文摘要 · Abstract (English)
The exponential increase in video content poses significant challenges in terms of efficient navigation, search, and retrieval, thus requiring advanced video summarization techniques. Existing video summarization methods, which heavily rely on visual features and temporal dynamics, often fail to capture the semantics of video content, resulting in incomplete or incoherent summaries. To tackle the challenge, we propose a new video summarization framework that leverages the capabilities of recent Large Language Models (LLMs), expecting that the knowledge learned from massive data enables LLMs to evaluate video frames in a manner that better aligns with diverse semantics and human judgments, effectively addressing the inherent subjectivity in defining keyframes. Our method, dubbed LLM-based Video Summarization (LLMVS), translates video frames into a sequence of captions using a Muti-modal Large Language Model (M-LLM) and then assesses the importance of each frame using an LLM, based on the captions in its local context. These local importance scores are refined through a global attention mechanism in the entire context of video captions, ensuring that our summaries effectively reflect both the details and the overarching narrative. Our experimental results demonstrate the superiority of the proposed method over existing ones in standard benchmarks, highlighting the potential of LLMs in the processing of multimedia content.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。