Aha实时预测视频关键帧,无需预览未来画面,适合机器人等实时场景。
AHA -- Predicting What Matters Next: Online Highlight Detection Without Looking Ahead

- 采用自回归框架与多模态模型,逐帧判断视频内容相关性。
- 在TVSum和Mr. Hisum上分别提升5.9%和8.3%的mAP,超越离线方法。
- 动态缓存机制实现无限长视频恒定内存,适合真实机器人部署。
实时理解连续视频流对自动驾驶、监控无人机和灾后救援机器人等高风险环境中的智能体至关重要。然而,现有视频理解与亮点检测方法通常依赖完整视频输入,不适用于在线或流式场景。当前模型多优化于离线摘要,难以支持实时决策所需的逐步推理。本文提出Aha,一种自回归亮点检测框架,能基于自然语言任务描述,逐帧预测视频帧的相关性,无需访问未来帧。Aha结合多模态视觉-语言模型与轻量级解耦头,在大规模人工标注的人类中心视频数据集上训练。为实现可扩展性,引入动态SinkCache机制,在无限长视频流中保持恒定内存占用,且不降低基准测试性能。该机制促使隐藏表示捕捉高层任务目标,从而有效进行信息量、相关性和不确定性排序。Aha在亮点检测基准上达到最新性能,超越先前离线全上下文方法及视频-语言模型,在TVSum和Mr. Hisum上的mAP分别提升5.9%和8.3%。实验验证其在任务导向自然语言输入与持续机器人视角视频下的应用潜力,证明其作为下游规划与长时程理解的实时推理模块的有效性。
原文摘要 · Abstract (English)
Real-time understanding of continuous video streams is essential for intelligent agents operating in high-stakes environments, including autonomous vehicles, surveillance drones, and disaster response robots. Yet, most existing video understanding and highlight detection methods assume access to the entire video during inference, making them unsuitable for online or streaming scenarios. In particular, current models optimize for offline summarization, failing to support step-by-step reasoning needed for real-time decision-making. We introduce Aha, an autoregressive highlight detection framework that predicts the relevance of each video frame against a task described in natural language. Without accessing future video frames, Aha utilizes a multimodal vision-language model and lightweight, decoupled heads trained on a large, curated dataset of human-centric video labels. To enable scalability, we introduce the Dynamic SinkCache mechanism that achieves constant memory usage across infinite-length streams without degrading performance on standard benchmarks. This encourages the hidden representation to capture high-level task objectives, enabling effective frame-level rankings for informativeness, relevance, and uncertainty with respect to the natural language task. Aha achieves state-of-the-art (SOTA) performance on highlight detection benchmarks, surpassing even prior offline, full-context approaches and video-language models by +5.9% on TVSum and +8.3% on Mr. Hisum in mAP (mean Average Precision). We explore Aha's potential for real-world robotics applications given a task-oriented natural language input and a continuous, robot-centric video. Both experiments demonstrate Aha's potential effectiveness as a real-time reasoning module for downstream planning and long-horizon understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。