工具能否提升效率，取决于模型是否真正会用且能整合。
Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use Agents
- 通过多轮强化学习优化工具调用行为，提升决策准确性
- 压缩图像上下文可减少30%输入量，准确率仅轻微下降
- 适合研究人机交互与智能代理的工具集成问题
混合式计算机使用代理可通过截图或调用文本工具执行任务。在相同OSWorld-MCP基准（309项任务）下，同一组工具使推理模型性能提升4.0个百分点，却使非推理模型下降5.9个百分点（各运行5次，均超过2倍标准误）。根本差异在于工具决策行为：非推理模型常忽略、误命名或错误终止工具调用。推理模型虽避免此类错误，但仅在55/309项任务中调用工具，占可调用任务的23.9%。这一差距称为采用缺口。根源在于模型已有更低成本路径，且未被训练去使用工具。密集工具奖励机制使电子表格使用率从0.03升至0.33，并延续至贪婪解码，但保留准确率未提升。行为可调控，能力却不可。瓶颈在于工具调用语义理解。在上下文层面，成功调用工具后下一帧截图常冗余。移除该帧并减半图像历史，使输入token减少约三分之一，准确率小幅下降。在相同观测规则下重训练后，该成本消失。压缩后的代理达到37.8%，优于未压缩版本的33.0%，输入成本仅为53%，并在注册的降级子集上完全弥合了资源丰富与稀缺之间的差距。工具只有在模型主动选择并有效整合时才真正有用，当前混合代理仍大量闲置此类机会。
原文摘要 · Abstract (English)
Hybrid computer-use agents can act through screenshots or call text tools. We find that having a tool available does not settle which way the effect goes. Under one identical GUI-MCP harness on the OSWorld-MCP benchmark (309 tasks), the same MCP tools improve a reasoning model by +4.0pp and degrade a non-reasoning model by -5.9pp (5 runs each, both beyond 2 SE). What separates the two is tool-decision behavior. The non-reasoning policy ignores, misnames, or falsely terminates around tools. The reasoning model avoids these failures, yet still calls a tool on only 55/309 tasks, 23.9% of the tool-reachable ones. We call this shortfall the adoption gap. Both levels of the problem share one cause: the model already has a cheaper route and is never trained to take it. Multi-turn RL probes that cause. At the action level, a dense tool bonus raises spreadsheet adoption 0.03 -> 0.33 and carries into greedy decoding, but held-out accuracy does not follow. Behavior is steerable; competence is not. The bottleneck lies in tool-call semantics. At the context level, a successful tool call often makes the next screenshot redundant. Dropping it and halving image history cuts input tokens by about a third, at a small accuracy cost. Retraining under the same observation rule removes that cost. The compressed agent then reaches 37.8% against 33.0% for the uncompressed operating point, at 53% of the input cost, and closes the rich-lean gap on a pre-registered degraded subset to zero. Tools help when the model chooses and integrates them, and current hybrid agents leave many such choices unused. Code and checkpoints: https://github.com/redai-infra/hybrid-routing-agent
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。