arXiv:2606.14476cs.AIcs.LG2026-06被引 1

大模型代理盲目听从图神经网络工具,能力越强越依赖,反而丢掉自主判断。

When the Tool Decides: LLM Agents Defer Blindly to Graph Neural Network Tools, and Stronger Backbones Defer More

  • 让大模型代理调用冻结的图神经网络工具,发现其几乎完全复制工具输出。
  • 模型越大,对工具的依赖越深,7B模型与工具一致率达98%。
  • 仅靠简单门控机制无法挽回损失,自主决策受限于可用信息而非设计缺陷。

越来越多工作将图神经网络(GNN)作为可调用工具赋予大语言模型(LLM)代理,假设代理能自主判断何时及多大程度依赖该工具。我们直接测试这一假设:在文本属性图(ogbn-arxiv,复现于WikiCS)上,将冻结的GNN作为显式工具提供给基于ReAct风格的LLM代理,测量其是否主动使用或仅仅服从工具。结果发现,代理并未行使判断力:其预测与原始GNN结果一致率达97.6%-99.2%(5次种子实验),表现如复读机般全盘接受工具输出,绕过自身推理。在不同骨干模型能力(Qwen2.5 0.5B-7B)下,这种盲从并非弱模型特例——在可调用工具的模型中,一致性随能力提升而上升(1.5B至7B从0.60升至0.98)。关键的是,随着模型能力增强,盲从代价不减反增,尤其在高同质性场景下更明显:一个基于每节点动作的最优解(oracle)在3B和7B时分别优于“复读机”0.09-0.18和0.12-0.22,且差距在高同质性下翻倍;在7B时,仅需邻居标签的简单工具已超越原GNN(0.81 vs 0.71),但代理仍持续盲从。引入简单选择性调用门控仅恢复高同质性下一半差距(0.71→0.83),全局无净收益;外部估计表明,仅凭标准测试时特征,最佳门控最多捕捉三分之一的最优解空间:可靠的智能调用受限于信息可用性,而非路由器设计本身。研究警示:评估代理+工具系统时不可默认代理具备额外判断力,选择性调用必须主动设计,不能寄望于规模增长自然涌现。

原文摘要 · Abstract (English)

A growing line of work equips large language model (LLM) agents with graph neural networks (GNNs) as callable tools, assuming the agent exercises judgment over when and how much to rely on such a tool. We test this directly. We expose a frozen GNN to a ReAct-style LLM agent as an explicit tool and measure, on node classification over a text-attributed graph (ogbn-arxiv, replicated on WikiCS), whether the agent uses the tool or merely obeys it. We find the agent does not exercise judgment: its predictions agree with the raw GNN's 97.6-99.2% of the time (5 seeds), collapsing into a GNN parrot that adopts the tool's output wholesale and bypasses its own reasoning. Sweeping backbone capability (Qwen2.5 0.5B-7B), the deference is not a weak-model artifact: among models able to invoke the tool, agreement rises with capability (0.60 to 0.98 from 1.5B to 7B). Crucially, the cost of deference does not shrink as capability grows and grows where alternatives emerge: a per-node oracle over the available actions beats the parrot by 0.09-0.18 at 3B and 0.12-0.22 at 7B, roughly doubling at high homophily, because the parrot is pinned to the frozen GNN while the agent's alternatives improve; at 7B a simple neighbour-label tool overtakes the GNN at high homophily (0.81 vs 0.71) yet the agent still defers. A simple selective-invocation gate recovers about half of that high-homophily gap (0.71 to 0.83) but yields no net global gain, and held-out estimates bound the best achievable gate over standard test-time features to at most a third of the oracle headroom: reliable selective invocation looks limited by available information, not merely router design. Our results are a cautionary measurement: evaluations of agent+tool systems cannot assume the agent adds judgment on top of the tool, and selective invocation must be designed in rather than expected to emerge from scale.

大模型代理图神经网络盲从风险选择性调用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。