让视频智能体重用视觉工具计算结果,提升长视频理解能力
LAVE: Latent Visual Evidence-Enhanced Planning for Video Tool-use Agents

- 用双通道接口保存工具调用的视觉证据,不依赖文字摘要
- 在有限帧预算下,视频任务准确率提升3.76分
- 无需训练或修改原系统,适合现有视频代理快速升级
长视频理解需要模型高效获取并复用冗长视频流中的稀疏视觉证据。当前视频工具使用智能体通过在不同时间尺度上迭代调用视觉工具来应对这一挑战,但其工具-规划器通信通常仅依赖文本观察。这种纯文本接口会丢失未被语言描述的视觉计算结果,导致已有视觉证据被丢弃,无法用于后续规划。我们识别出此为工具观察瓶颈,并提出无需训练的潜空间视觉证据增强规划框架(LAVE)。LAVE采用双通道观察接口:可见通道保留原始文本轨迹,潜空间通道存储预语言化的视觉更新,包含工具角色、源帧时间戳和视觉位置。规划阶段,LAVE检索当前规划状态中未被文本覆盖的相关证据,并通过时间对齐的潜空间更新与熵约束的帧时间路由进行整合。该方法使视频智能体可在不增加训练、不重放帧、不修改原有调度逻辑的前提下,复用已有的视觉计算。在Video-MME、LongVideoBench和CG-Bench上的大量实验表明,LAVE在不同骨干模型上均能持续提升性能。在相近帧预算下,相比最强基线,其在Video-MME总分上提升3.76分,验证了潜空间视觉证据重用在多步视频智能体规划中的有效性。
原文摘要 · Abstract (English)
Long-video understanding requires models to efficiently acquire and reuse sparse visual evidence from long and redundant video streams. Recent video tool-use agents address this challenge by iteratively invoking visual Tools at different temporal scales, but their Tool-Planner communication typically relies on textual observations. Such text-only interfaces provide lossy summaries of Tool computations, causing previously computed visual evidence not verbalized to be discarded and unavailable for subsequent planning. We identify this limitation as the Tool observation bottleneck and propose Latent Visual Evidence-Enhanced Planning (LAVE), a training-free framework for reusing latent visual evidence from completed Tool calls. LAVE introduces a dual-channel observation interface: the visible channel preserves the original textual trajectory, while the latent channel stores pre-verbal visual updates with their Tool roles, source-frame timestamps, and visual locations. During planning, LAVE retrieves evidence relevant to the current Planner state but not covered by textual observations, and integrates it through bounded timestamp-aligned latent updates with entropy-constrained frame-time routing. This enables video agents to reuse existing visual computation without additional training, frame replay, or modifications to the original orchestration. Extensive experiments on Video-MME, LongVideoBench, and CG-Bench show that LAVE consistently improves video tool-use agents across backbones. Under a comparable frame budget, LAVE improves the Video-MME overall score by 3.76 points over the strongest baseline, demonstrating the effectiveness of latent visual evidence reuse for multi-step video-agent planning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。