让智能体学会何时用工具,避免盲目调用。
Act Wisely: Cultivating Meta-Cognitive Tool Use in Agentic Multimodal Models
- 用条件优势估计分离准确率与效率目标
- 工具调用次数减少数量级,推理准确率提升
- 适合需要高效决策的多模态智能体系统
代理型多模态模型虽能主动与外部环境交互,但普遍存在元认知缺陷:难以权衡使用内部知识还是调用外部工具。当前系统常盲目调用工具,即使问题可从视觉上下文中直接解决,导致严重延迟和推理噪声。现有强化学习方法通过惩罚工具使用的标量奖励缓解此问题,但存在优化困境——严厉惩罚抑制必要调用,轻微惩罚则被准确率奖励方差淹没。为此,本文提出HDPO框架,将工具效率从竞争性标量目标转为严格条件化目标。通过避免奖励标量化,HDPO维持两个正交优化通道:准确率通道最大化任务正确性,效率通道仅在准确轨迹中强制执行经济性。该解耦结构自然形成认知训练路径,促使代理先掌握任务求解再优化自立性。大量实验表明,所提模型Metis使工具调用减少数量级,同时提升推理准确率。
原文摘要 · Abstract (English)
The advent of agentic multimodal models has empowered systems to actively interact with external environments. However, current agents suffer from a profound meta-cognitive deficit: they struggle to arbitrate between leveraging internal knowledge and querying external utilities. Consequently, they frequently fall prey to blind tool invocation, resorting to reflexive tool execution even when queries are resolvable from the raw visual context. This pathological behavior precipitates severe latency bottlenecks and injects extraneous noise that derails sound reasoning. Existing reinforcement learning protocols attempt to mitigate this via a scalarized reward that penalizes tool usage. Yet, this coupled formulation creates an irreconcilable optimization dilemma: an aggressive penalty suppresses essential tool use, whereas a mild penalty is entirely subsumed by the variance of the accuracy reward during advantage normalization, rendering it impotent against tool overuse. To transcend this bottleneck, we propose HDPO, a framework that reframes tool efficiency from a competing scalar objective to a strictly conditional one. By eschewing reward scalarization, HDPO maintains two orthogonal optimization channels: an accuracy channel that maximizes task correctness, and an efficiency channel that enforces execution economy exclusively within accurate trajectories via conditional advantage estimation. This decoupled architecture naturally induces a cognitive curriculum-compelling the agent to first master task resolution before refining its self-reliance. Extensive evaluations demonstrate that our resulting model, Metis, reduces tool invocations by orders of magnitude while simultaneously elevating reasoning accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。