首个面向第一人称视频个性化问答的评测数据集，揭示大模型在理解‘自我’上的严重不足。
Ego-Grounding for Personalized Question-Answering in Egocentric Videos
- 构建首个第一人称视频个性化问答数据集MyEgo，聚焦对‘我’的认知与记忆。
- 顶尖模型准确率仅46%（如GPT-5），远低于人类，且模型规模和推理能力无法稳定提升性能。
- 关键瓶颈在于长期追踪和记忆‘自我’，证据提供虽短期有效但随时间衰减。
我们首次系统分析了多模态大语言模型（MLLMs）在需要第一人称定位（ego-grounding）的个性化问答任务中的表现——即理解第一人称视频中摄像头佩戴者的能力。为此，我们提出了MyEgo，首个专为评估MLLMs对摄像头佩戴者理解、记忆与推理能力而设计的第一人称视频问答数据集。MyEgo包含541段长视频和5000个个性化问题，涵盖“我的物品”“我的活动”和“我的过去”。基准测试显示，各类先进模型（包括开源与闭源、有无思维链、大小模型）在MyEgo上均表现不佳。顶级闭源模型（如GPT-5）与开源模型（如Qwen3-VL）准确率分别仅为~46%和36%，相比人类分别落后近40%和50%。令人意外的是，显式推理或模型扩容并未带来一致性能提升。当相关证据被明确提供时，模型性能改善，但该增益随时间衰减，表明其在追踪和记忆“我”及“我的过去”方面存在根本局限。这些发现共同凸显了第一人称定位与长程记忆在实现第一人称个性化辅助中的关键作用。我们希望MyEgo及其分析能推动该领域进一步进展。数据与代码已公开于https://github.com/Ryougetsu3606/MyEgo。
原文摘要 · Abstract (English)
We present the first systematic analysis of multimodal large language models (MLLMs) in personalized question-answering requiring ego-grounding - the ability to understand the camera-wearer in egocentric videos. To this end, we introduce MyEgo, the first egocentric VideoQA dataset designed to evaluate MLLMs' ability to understand, remember, and reason about the camera wearer. MyEgo comprises 541 long videos and 5K personalized questions asking about "my things", "my activities", and "my past". Benchmarking reveals that competitive MLLMs across variants, including open-source vs. proprietary, thinking vs. non-thinking, small vs. large scales all struggle on MyEgo. Top closed- and open-source models (e.g., GPT-5 and Qwen3-VL) achieve only~46% and 36% accuracy, trailing human performance by near 40% and 50% respectively. Surprisingly, neither explicit reasoning nor model scaling yield consistent improvements. Models improve when relevant evidence is explicitly provided, but gains drop over time, indicating limitations in tracking and remembering "me" and "my past". These findings collectively highlight the crucial role of ego-grounding and long-range memory in enabling personalized QA in egocentric videos. We hope MyEgo and our analyses catalyze further progress in these areas for egocentric personalized assistance. Data and code are available at https://github.com/Ryougetsu3606/MyEgo
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。