让AI像人一样分步推理超长第一视角视频，突破时间跨度限制。
Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning
- 用分步工具链思维模拟人类解题，每步调用专用工具处理子任务。
- 在周级视频上实现精准问答，时间覆盖从数小时扩展至整周。
- 适合需要长时序理解的视频分析、智能助理等场景。
我们提出Ego-R1，一种针对超长（长达数天甚至数周）第一视角视频的新型推理框架，通过由强化学习训练的Ego-R1 Agent驱动的结构化工具链思维（CoTT）流程实现。受人类问题解决策略启发，CoTT将复杂推理分解为模块化步骤，每一步由强化学习代理调用特定工具，协同完成时间检索与多模态理解等任务。采用两阶段训练范式：先用包含2.5万条CoTT数据的Ego-CoTT-25K进行监督微调，再利用4.4千条Ego-QA-4.4K数据进行强化学习，使代理能动态规划逐步推理路径。为支持训练，构建了Ego-R1 Data数据集。此外，在新创建的周级视频问答基准Ego-R1 Bench上评估，该基准包含来自混合源的人工验证问答对。大量实验表明，该框架能有效应对超长第一视角视频的理解挑战，显著将时间覆盖范围从数小时扩展至一周。
原文摘要 · Abstract (English)
We introduce Ego-R1, a novel framework for reasoning over ultra-long (i.e., in days and weeks) egocentric videos, which leverages a structured Chain-of-Tool-Thought (CoTT) process, orchestrated by an Ego-R1 Agent trained via reinforcement learning (RL). Inspired by human problem-solving strategies, CoTT decomposes complex reasoning into modular steps, with the RL agent invoking specific tools, one per step, to iteratively and collaboratively answer sub-questions tackling such tasks as temporal retrieval and multi-modal understanding. We design a two-stage training paradigm involving supervised finetuning (SFT) of a pretrained language model using CoTT data and RL to enable our agent to dynamically propose step-by-step tools for long-range reasoning. To facilitate training, we construct a dataset called Ego-R1 Data, which consists of Ego-CoTT-25K for SFT and Ego-QA-4.4K for RL. Furthermore, our Ego-R1 agent is evaluated on a newly curated week-long video QA benchmark, Ego-R1 Bench, which contains human-verified QA pairs from hybrid sources. Extensive results demonstrate that the dynamic, tool-augmented chain-of-thought reasoning by our Ego-R1 Agent can effectively tackle the unique challenges of understanding ultra-long egocentric videos, significantly extending the time coverage from few hours to a week.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。