构建首个第一人称视频陪伴式问答基准，推动智能助手理解实时情境。
Companion-style QA Assistance in Ego-Vision
- 提出多模态思维链机制，结合历史问答与视觉流推理意图
- 在21.6K问题上实现显著性能提升，跨数据集泛化有效
- 适合研究视频理解、交互式智能助手的开发者与学者
AI伴侣被设想为全天候支持日常生活的助手。为此，我们提出BuddyVQA，一个针对第一人称视频流的陪伴式问答基准。该数据集包含21.6K个问题，关联6,000个关键片段，覆盖1,012段长时第一人称视频。其核心特征包括第一人称指示表达（如“它在哪？”）和交互式连环提问（如“怎么去那里？”），要求模型在长视频流中基于上下文解析视觉代词并推断用户意图。为此，我们提出MyBuddy，一种具备多模态思维链推理能力的陪伴式问答助手，通过历史问答与视觉内容联合建模生成答案，并引入问题过滤器与多级记忆模块，提升流式问答下的信息检索效率。实验表明，MyBuddy显著提升基础模型在BuddyVQA上的表现，且效果可推广至其他流式与常规视频问答数据集，验证了方法的有效性与普适性。代码与数据集已开源。
原文摘要 · Abstract (English)
AI companions are envisioned as always-on assistants that support users in daily life. With this regard, we introduce BuddyVQA, a benchmark for companion-style question answering (QA) on egocentric streaming video. BuddyVQA contains 21.6K questions linked to 6K highlight moments across 1,012 long, egocentric videos. It features two key characteristics that are common in daily first-person QA assistance but are largely overlooked in existing VideoQA benchmarks: ego-deictic expressions and interactively chained questions (e.g., "Where is it?", "How to get there?"). These require models to infer a user's in-situation intent by resolving visual pronouns in the context of egocentric visual and QA contents, with both grounded in a long-form streaming setting. To tackle the challenges, we propose MyBuddy, a companion-style QA assistant that highlights a multimodal chain-of-thought reasoning mechanism to infer the final answer based on the historical QA and visual content. An additional question filter and multi-level memory are designed to facilitate efficient QA and visual information retrieval under streaming QA settings. Experiments show that MyBuddy significantly enhances the performance of foundation models on BuddyVQA. Moreover, these gains generalize to other streaming and common video QA benchmarks, demonstrating the applicability and effectiveness of our approach. Our code and dataset are available at https://github.com/QHUni/BuddyVQA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。