arXiv:2607.17994cs.CVcs.AI2026-07

用亮点引导注意力,让多模态大模型更精准地总结视频内容。

HAS: Highlight-guided Attention Steering for Multimodal LLM Video Summarization

论文配图:HAS: Highlight-guided Attention Steering for Multimodal LLM Video Summarization
图 1 · 摘自论文原文
  • 通过全局连续亮点分布引导模型注意力
  • 在多个基准上实现显著更好的摘要效果
  • 适合需要精准视频理解的场景,如智能检索

随着AI视频生成技术的发展,视频理解日益重要。多模态大语言模型(M-LLM)在视频理解方面展现出强大能力,而视频摘要作为其关键应用,对高效导航与检索具有重要意义。现有方法多聚焦于选取离散的关键帧及其关联段落描述,但忽视了从全局视角评估帧的重要性。我们指出,仅依赖离散关键帧会降低理解连贯性,丢失视频中重要信息,并浪费M-LLM的原始能力。为此,本文提出HAS(Highlight-guided Attention Steering),一种基于亮点引导的注意力调控方法。该方法包含两部分:一是全局生成连续的帧级亮点分布;二是将该分布作为注意力权重向量输入给M-LLM,使模型在推理时更关注亮点帧,同时通过降低非亮点帧的关注度而非遗忘,保留完整信息。我们在多个基准数据集上验证了HAS的有效性,结果表明其在视频摘要任务中表现优异。

原文摘要 · Abstract (English)

Video understanding has become more and more important with the growth of Artificial Intelligence (AI) for video generation. Recently, Multimodal Large Language Model(M-LLM) has shown its capability in video understanding. Video summarization, a specific domain of video understanding, has proven its importance for efficient navigation and retrieval. Both video understanding and video summarization require a good selection of key frames in a video. Current video summarization methods heavily focus on the selected key frames and correlated segment captions. However, existing approaches overlook the perspective of treating the importance of the frames globally. We argue that using discrete selected frames for summarization will not only reduce the understanding coherence, but also lost important information in the video, as well as wasting the original capacity of the MLLMs. In this paper, we propose HAS, a Highlight-guided Attention Steering method for video summarization. We consider a challenging but practical setting where the video given to MLLMs for summarize should be continuous but with highlight guidance. HAS mainly consists of two parts: The first part is to find a continuous frame-level highlight distribution for the video globally. The second part is to apply the highlight distribution as an attention steering vector for the MLLM, targeting a better understanding of the video, and thus during the model inference time, putting more attention on the highlighted frames, while avoiding lost entire information on less highlighted frames through putting less attention instead of forgetting them. We evaluated HAS on a variety of benchmarks, and it has shown convincing performance in video summarization.

视频摘要多模态模型注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。