arXiv:2510.00307cs.AI2025-10被引 5

发现大模型选工具有偏见,提出方法减轻不公平倾向。

BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language Models

  • 构建多类别等效工具基准,评估模型选择偏好。
  • 模型倾向固定选某厂商或靠前出现的工具,偏差显著。
  • 优化提示和随机采样可有效降低偏见,适合开发者参考。

由大语言模型驱动的智能体越来越多地依赖来自市场中的多个功能等效外部工具。这引发了一个关键公平性问题:工具选择中的系统性偏见会损害用户体验,并扭曲竞争,使某些提供方获得不正当优势。本文构建了一个涵盖多种工具类别的基准,每类包含多个功能等效工具,用于系统评估工具选择偏见。基于该基准,我们评估了七种大语言模型,发现显著偏见仍然存在,模型要么固守单一提供方,要么明显偏好上下文较早出现的工具。通过受控实验,我们隔离分析了工具特征、暴露元数据(名称、描述、参数)以及预训练暴露的影响,发现:(1)用户查询与工具元数据之间的语义对齐是选择行为最强驱动力;(2)工具描述的微小改动可显著改变选择结果;(3)对单一端点的重复预训练暴露会加剧提供方级偏见。最后,我们提出一种轻量级缓解策略:先过滤出相关工具子集,再进行均匀采样,在大幅降低选择偏见的同时保持良好的任务覆盖。结果表明,工具选择偏见是工具增强型大模型代理公平部署的关键障碍。代码与基准已公开于 https://github.com/thierry123454/tool-selection-bias。

原文摘要 · Abstract (English)

Agents backed by large language models (LLMs) increasingly rely on external tools drawn from marketplaces where multiple providers offer functionally equivalent options. This raises a critical fairness concern: systematic bias in tool selection can degrade user experience and distort competition by privileging certain providers over others. We introduce a benchmark of diverse tool categories, each containing multiple functionally equivalent tools, to systematically evaluate tool-selection bias. Using this benchmark, we evaluate seven LLMs and show that substantial bias persists, with models either fixating on a single provider or disproportionately favoring tools that appear earlier in the context. To uncover the sources of this behavior, we conduct controlled experiments that isolate the effects of tool features, exposed metadata (name, description, and parameters), and pre-training exposure. We find that (1) semantic alignment between user queries and tool metadata is the strongest driver of selection; (2) small perturbations to tool descriptions can significantly shift choices; and (3) repeated pre-training exposure to a single endpoint amplifies provider-level bias. Finally, we propose a lightweight mitigation strategy that first filters tools to a relevant subset and then samples uniformly, substantially reducing selection bias while maintaining strong task coverage. Our results highlight tool-selection bias as a key obstacle to the fair deployment of tool-augmented LLM agents. Our code and benchmark are publicly available at https://github.com/thierry123454/tool-selection-bias.

大模型工具选择偏见缓解公平性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。