让大模型学会第一人称视角推理，精准理解动作与物品关系。
EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT
- 用时空思维链监督训练，提升模型对隐藏意图的推断能力。
- 在1300万视频片段上构建500万条问答数据，支持细粒度推理。
- 适合研究视觉-语言模型、人机交互及第一人称视频分析的开发者。
第一人称视频推理的核心挑战在于，摄像头背后不可见的主体动态影响环境，需推断其隐含意图并识别精细交互。现有多模态大模型虽擅长可见事件推理，但缺乏具身化、第一人称理解能力。为此，我们提出EgoThinker框架，通过时空思维链监督和两阶段学习课程，赋予大模型鲁棒的第一人称推理能力。首先，我们构建了EgoRe-5M，一个大规模第一人称问答数据集，基于1300万段多样化第一人称视频剪辑，包含多分钟片段，标注有详细的思维链推理过程和密集的手物定位信息。其次，我们在EgoRe-5M上进行监督微调（SFT），注入推理能力，并通过强化学习微调（RFT）进一步提升时空定位精度。实验表明，EgoThinker在多个第一人称基准测试中超越现有方法，在细粒度时空定位任务上取得显著提升。完整代码与数据已开源至https://github.com/InternRobotics/EgoThinker。
原文摘要 · Abstract (English)
Egocentric video reasoning centers on an unobservable agent behind the camera who dynamically shapes the environment, requiring inference of hidden intentions and recognition of fine-grained interactions. This core challenge limits current multimodal large language models MLLMs, which excel at visible event reasoning but lack embodied, first-person understanding. To bridge this gap, we introduce EgoThinker, a novel framework that endows MLLMs with robust egocentric reasoning capabilities through spatio-temporal chain-of-thought supervision and a two-stage learning curriculum. First, we introduce EgoRe-5M, a large-scale egocentric QA dataset constructed from 13M diverse egocentric video clips. This dataset features multi-minute segments annotated with detailed CoT rationales and dense hand-object grounding. Second, we employ SFT on EgoRe-5M to instill reasoning skills, followed by reinforcement fine-tuning RFT to further enhance spatio-temporal localization. Experimental results show that EgoThinker outperforms existing methods across multiple egocentric benchmarks, while achieving substantial improvements in fine-grained spatio-temporal localization tasks. Full code and data are released at https://github.com/InternRobotics/EgoThinker.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。