arXiv:2605.18176cs.CVcs.AI2026-05被引 1

MARS通过多模态证据选择,在复杂长时视频任务中取得第二名。

MARS: Technical Report for the CASTLE Challenge at EgoVis 2026

  • 构建多模态证据库,整合视频、字幕与生理信号等8类数据
  • 采用决策代理动态选择证据或请求缺失模态,推理准确率领先
  • 适合需要长时序多源信息融合的智能助手研究者参考

本报告介绍MARS(Multimodal Agentic Reasoning with Source selection),即多模态代理推理与源选择系统,用于参加EgoVis 2026的CASTLE挑战赛。参赛者需回答185个封闭式问题,基于CASTLE 2024数据集。该数据集涵盖四天活动、15个同步视角、官方字幕及多种辅助模态:个人照片、辅助视频、注视追踪、热成像与心率数据。与以往单视频自指基准不同,CASTLE要求跨模态长期推理。MARS将任务视为多源证据选择的代理问题,首先按官方目录结构,从视频与字幕两大主源,以及注视、心率、照片、热成像四个辅源构建证据记忆。由于视频过长无法直接输入模型,将其转换为字幕与基于DeepSeek的摘要,以压缩时间证据;同时保留照片及其他辅助媒体作为特定源证据。推理阶段,使用GPT-5.4决策代理反复判断是否继续推理、请求特定缺失模态、生成答案,或在证据不足时随机作答。该系统在最终排行榜上获得第二名。代码已开源:https://github.com/Hyu-Zhang/MARS。

原文摘要 · Abstract (English)

This report presents MARS, short for Multimodal Agentic Reasoning with Source selection, our system for the CASTLE Challenge at EgoVis 2026. Participants must answer 185 closed-form questions over the CASTLE 2024 dataset. In contrast to prior single-video egocentric benchmarks, CASTLE requires reasoning over four days of activity, 15 synchronized perspectives, official transcripts, and multiple auxiliary modalities, including personal photos, auxiliary videos, gaze, thermal imagery, and heartrate measurements. MARS therefore treats the task as an agentic evidence-selection problem over multimodal sources rather than a purely text-only pipeline. MARS first follows the official CASTLE directory organization to build evidence memories from two primary sources, videos and transcripts, and four auxiliary sources, gaze, heartrate, photos, and thermal imagery. Long videos are converted into captions and DeepSeek-based summaries only because CASTLE videos are too long to fit directly into the model context for every question; this step compresses temporal evidence while keeping photos and other auxiliary media available as source-specific evidence. At inference time, a GPT-5.4 decision agent repeatedly chooses whether to continue reasoning, request a specific missing modality, produce an answer, or fall back to a random option when the evidence remains insufficient. The resulting system achieved second place on the final CASTLE Challenge leaderboard. Our codes are available at https://github.com/Hyu-Zhang/MARS.

多模态长时推理智能代理自指视觉

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。