将长视频分解为三类语义实体，用注意力机制高效压缩冗余信息。
SEAL: Semantic Attention Learning for Long Video Representation
- 按场景、物体、动作拆分视频，减少计算量
- 注意力模块提升关键帧选择效率，跨任务表现更优
- 适合需要理解长视频语义的任务，如问答与定位
长视频理解面临计算复杂度高和时间冗余的挑战。有效的长视频表示需在处理冗余的同时保留下游任务所需关键内容。本文提出一种统一的长视频表示方法SEAL，通过将长视频分解为场景、物体和动作三类语义实体，使模型可在少量实体上操作，而非大量帧或像素。为进一步缓解冗余，提出一种注意力学习模块，以平衡标记相关性与多样性，将其建模为子集选择优化问题。该表示具有通用性，适用于多种长视频理解任务。大量实验表明，SEAL在多个基准（包括LVBench、MovieChat-1K和Ego4D）上的视频问答和时序定位任务中显著优于现有方法。
原文摘要 · Abstract (English)
Long video understanding presents challenges due to the inherent high computational complexity and redundant temporal information. An effective representation for long videos must efficiently process such redundancy while preserving essential contents for downstream tasks. This paper introduces SEmantic Attention Learning (SEAL), a novel unified representation for long videos. To reduce computational complexity, long videos are decomposed into three distinct types of semantic entities: scenes, objects, and actions, allowing models to operate on a compact set of entities rather than a large number of frames or pixels. To further address redundancy, we propose an attention learning module that balances token relevance with diversity, formulated as a subset selection optimization problem. Our representation is versatile and applicable across various long video understanding tasks. Extensive experiments demonstrate that SEAL significantly outperforms state-of-the-art methods in video question answering and temporal grounding tasks across diverse benchmarks, including LVBench, MovieChat-1K, and Ego4D.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。