让大模型根据用户和环境智能选工具,提升使用体验。
ToolSpectrum : Towards Personalized Tool Utilization for Large Language Models
- 构建双维度个性化评估框架,融合用户画像与环境因素。
- 实验证明个性化选工具可显著提升用户体验,但现模型能力有限。
- 适合研究工具增强型大模型、个性化交互的开发者与研究者。
将外部工具集成到大语言模型(LLMs)中可增强其获取实时信息和特定领域服务的能力,但现有方法仅关注根据用户指令选择功能工具,忽视了上下文感知的个性化选工具。这一疏漏导致用户满意度下降和工具利用效率低下,尤其在工具集重叠时更显突出。为此,我们提出ToolSpectrum基准,用于评估LLMs在个性化工具使用方面的能力。具体地,我们形式化了两个关键个性化维度:用户画像与环境因素,并分析它们各自及协同作用对工具选择的影响。在ToolSpectrum上的大量实验表明,个性化工具使用能显著提升多样场景下的用户体验。然而,即使最先进的LLMs也表现出有限的联合推理能力,常在用户画像与环境因素之间权衡失当。研究结果强调了上下文感知个性化的重要性,并揭示了当前模型的关键局限性。数据与代码已开源:https://github.com/Chengziha0/ToolSpectrum。
原文摘要 · Abstract (English)
While integrating external tools into large language models (LLMs) enhances their ability to access real-time information and domain-specific services, existing approaches focus narrowly on functional tool selection following user instructions, overlooking the context-aware personalization in tool selection. This oversight leads to suboptimal user satisfaction and inefficient tool utilization, particularly when overlapping toolsets require nuanced selection based on contextual factors. To bridge this gap, we introduce ToolSpectrum, a benchmark designed to evaluate LLMs' capabilities in personalized tool utilization. Specifically, we formalize two key dimensions of personalization, user profile and environmental factors, and analyze their individual and synergistic impacts on tool utilization. Through extensive experiments on ToolSpectrum, we demonstrate that personalized tool utilization significantly improves user experience across diverse scenarios. However, even state-of-the-art LLMs exhibit the limited ability to reason jointly about user profiles and environmental factors, often prioritizing one dimension at the expense of the other. Our findings underscore the necessity of context-aware personalization in tool-augmented LLMs and reveal critical limitations for current models. Our data and code are available at https://github.com/Chengziha0/ToolSpectrum.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。