让多模态问答有迹可循,自动追踪证据是否齐备
Omni-Decision: A Progressive Evidence-State Agent System for Omni-Modal QA

- 用结构化状态跟踪证据、矛盾和待补信息
- 在OmniGAIA上达45.6%准确率,提升27.3个百分点
- 适合需要可追溯推理的复杂多模态任务
多模态证据搜索型问答需从视频、音频、图像、网页和计算结果中提取分散证据。现有代理系统常将证据存于临时缓存或自由文本历史中,难以追踪已确认、缺失或足够的证据。我们提出Omni-Decision,一种无需训练的证据状态系统,将多模态问答转化为有查询范围的证据闭合过程。针对每个问题,系统维护包含已确认证据、未解矛盾、事实与计算依赖、待补需求的结构化证据状态。共享状态视图指导规划、证据获取、验证、修复与最终决策。来自媒体、网络、计算与验证模块的异构观测经统一归一化、判断并确定性地写入状态。该设计实现精准证据获取,保留稀疏跨模态线索,并提供可审查的修复与终止控制。Omni-Decision在OmniGAIA上达到45.6%准确率,在WorldSense上达58.3%,分别优于基线+27.3和+30.2个百分点。无状态消融与轨迹审计进一步证实显式证据状态控制在多步多模态证据搜寻中的关键作用。
原文摘要 · Abstract (English)
Omni-modal evidence-seeking QA requires agents to answer questions whose evidence is sparsely distributed across videos, audio, images, web pages, and computation results. Existing agentic multimodal systems often leave evidence in scratchpads, tool trajectories, or free-form histories, making it difficult to track what has been grounded, what remains missing, and when the evidence is sufficient to answer. We propose Omni-Decision, a training-free evidence-state system that turns omni-modal QA into a query-scoped evidence-closure process. For each query, Omni-Decision maintains a structured evidence state containing confirmed evidence, unresolved conflicts, fact and computation dependencies, and open evidence needs. A shared state view conditions planning, evidence acquisition, validation, repair, and finalization. Heterogeneous observations from media, web, computation, and verification modules are normalized, judged, and committed through deterministic state updates. This design enables targeted evidence acquisition, preserves sparse cross-modal cues, and provides inspectable control over repair and stopping. Omni-Decision achieves 45.6% accuracy on OmniGAIA and 58.3% on WorldSense, improving over the baselines by +27.3 and +30.2 percentage points, respectively. No-state ablations and trajectory audits further support the role of explicit evidence-state control in multi-step omni-modal evidence seeking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。