构建用户偏好规划新基准,提升智能体个性化决策能力
SimGuide: Typed Multi-Context User Representations for Preference-Conditioned Agent Planning

- 用类型化上下文块与冲突仲裁机制,结构化表示用户偏好
- 在47个任务中,比记忆过往行为提升0.210~0.144的偏好遵循度
- 支持可追溯验证,适合研究个性化智能体与决策系统
代理为用户代行任务时,需根据用户差异调整规划策略,越来越多地依赖结构化用户上下文表示而非原始交互历史。但这种结构化表示的价值及其关键部分尚无量化评估。本文提出SimBench,包含47个基于9个合成用户的偏好条件规划任务,每个用户由28个类型化、可能冲突的上下文块构成,正确规划取决于哪些上下文激活及冲突如何解决。在此基准上评估SimGuide框架,该框架结合类型化多上下文表示、显式冲突仲裁及可选的约束过程化引导。在三个模型(Llama 3.3 70B、GPT-4o、Claude Sonnet 4.5)和六种用户-上下文表示中,带仲裁的类型化块相较检索同一用户过往决策,分别提升0.210、0.205和0.144的偏好遵循度(均p < 0.001);移除仲裁指令导致最高下降0.209。通过示例引导约束仅在模型有提升空间时有效:对Llama 70B提升+0.094(p < 0.001),对GPT-4o仅+0.029,对Claude几乎无增益(+0.002),后者已在35/47任务中完美表现。报告了基准的最小可检测效应,并提供溯源审计,确保所有结果可从生成它的提示中复现。
原文摘要 · Abstract (English)
Agents that act on a user's behalf must plan differently for different users, and increasingly do so from some structured representation of user context and not from raw interaction history. How much that structure is worth, and which parts of it carry the value, is largely unmeasured. We introduce SimBench, 47 preference-conditioned planning tasks over 9 synthetic users represented as 28 typed, potentially conflicting context blocks, where the correct plan depends on which contexts are active and how their conflicts are resolved. Against it we evaluate SimGuide, a framework combining typed multi-context representation, explicit conflict arbitration, and optional procedural grounding of individual constraints. Across three models and six user-context representations, SimGuide's typed blocks with arbitration outperform retrieval over the same user's past decisions by +0.210, +0.205 and +0.144 Preference Adherence on Llama 3.3 70B, GPT-4o and Claude Sonnet 4.5 respectively (all p < 0.001); removing the arbitration instruction alone costs up to +0.209. Grounding each constraint with a worked example of past application helps only where the model has headroom: +0.094 on Llama 70B (p < 0.001), falling to +0.029 on GPT-4o and +0.002 on Claude, which already scores perfectly on 35 of 47 tasks without it. We report the benchmark's minimum detectable effect alongside its results. The benchmark ships with a provenance audit that re-derives every reported number from the prompt that produced it.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。