arXiv:2603.23483cs.CVcs.CL2026-03被引 1

用轻量模型预判大模型任务路径,加速多模态智能体推理。

SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning

  • 轻量模型充当预判器,提前规划工具调用流程。
  • 实验显示速度提升1.1至3.35倍,准确率最高提升6.7%。
  • 适合高并发场景下需要低延迟的多模态智能体系统。

基于迭代视觉工具调用的智能体多模态大模型(如OpenAI o3和Gemini Agentic Vision)虽具备强大推理能力,但感知、推理与工具调用的串行循环带来显著延迟,即‘智能体深度’问题,严重制约系统吞吐。为此,我们提出SpecEyes——一种面向智能体层级的推测加速框架,突破此串行瓶颈。核心思路是利用一个轻量、无工具依赖的MLLM作为推测规划器,预测执行轨迹,实现对昂贵工具链的早期终止而不损失精度。为调控该推测过程,我们引入基于答案可分离性的认知门控机制,量化模型自验证信心,无需人工标签。此外,设计异构并行漏斗结构,利用小模型无状态并行性掩盖大模型有状态串行执行,最大化系统吞吐。在V* Bench、HR-Bench和POPE上的大量实验表明,SpecEyes相较基线实现1.1–3.35倍加速,同时保持或提升准确率(最高+6.7%),显著提升并发负载下的服务吞吐。

原文摘要 · Abstract (English)

Agentic multimodal large language models (MLLMs) (e.g., OpenAI o3 and Gemini Agentic Vision) achieve remarkable reasoning capabilities through iterative visual tool invocation. However, the cascaded perception, reasoning, and tool-calling loops introduce significant sequential overhead. This overhead, termed agentic depth, incurs prohibitive latency and seriously limits system-level concurrency. To this end, we propose SpecEyes, an agentic-level speculative acceleration framework that breaks this sequential bottleneck. Our key insight is that a lightweight, tool-free MLLM can serve as a speculative planner to predict the execution trajectory, enabling early termination of expensive tool chains without sacrificing accuracy. To regulate this speculative planning, we introduce a cognitive gating mechanism based on answer separability, which quantifies the model's confidence for self-verification without requiring oracle labels. Furthermore, we design a heterogeneous parallel funnel that exploits the stateless concurrency of the small model to mask the stateful serial execution of the large model, maximizing system throughput. Extensive experiments on V* Bench, HR-Bench, and POPE demonstrate that SpecEyes achieves 1.1-3.35x speedup over the agentic baseline while preserving or even improving accuracy (up to +6.7%), thereby boosting serving throughput under concurrent workloads.

多模态智能体加速推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。