测试大模型代理在模糊指令下的个性化能力，发现其依赖历史信息的能力不足。
APeB: Benchmarking Personalization Ability of Large Language Model Agents
- 构建新基准APeB，模拟用户未明确表达需求的真实场景
- 模型对明确指令表现好，但对初期模糊查询的意图识别准确率低
- 引入简单历史感知优化后性能显著提升，提示需专门设计历史利用模块
基于大语言模型的智能体在用户提出原始且不明确的查询时，难以实现个性化。此时，智能体必须推断潜在意图、从嘈杂的交互历史中提取偏好，并在多个候选方案间做出选择。现有评测基准通常依赖已优化的查询或简化的历史数据，难以真实检验该能力。本文提出个性化产品搜索（PPS）任务，作为在原始查询与多样化历史背景下的智能体个性化测试平台。基于真实操作日志构建了代理个性化基准（APeB），将不明确的意图与丰富的历史记录及用户浏览过的候选项配对。评估主流大模型在多步智能体工作流中的表现，发现模型虽能处理明确指令，但在需要早期意图和偏好发现的查询上表现不佳。评分分析表明，该差距主要源于对历史信息的无效利用。引入一个简单的历史感知查询优化流程VQRA后，性能获得持续提升，凸显出为个性化智能体专门设计历史利用模块的重要性。
原文摘要 · Abstract (English)
LLM-powered agents struggle with personalization when users issue raw, underspecified queries. In this setting, agents must infer latent intent, extract preferences from noisy interaction histories, and select among competing alternatives. Existing benchmarks rarely test this capability, as they often rely on user-refined queries or simplified histories. We introduce personalized product search (PPS), a testbed for agentic personalization under raw queries and diverse histories. We construct Agent Personalized Benchmark (APeB) from action logs, pairing underspecified intents with rich histories and user-viewed candidate items. Evaluating state-of-the-art LLMs with multi-step agent workflows, we find that models handle explicit queries well but struggle with early-stage queries requiring intent and preference discovery. Rubric analysis attributes this gap mainly to ineffective history use. A simple history-aware query-refinement pipeline, VQRA, yields consistent gains, highlighting the need for dedicated history-utilization modules in personalized agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。