给智能体技能做因果评估,精准淘汰有害技能,提升任务完成率。
Not All Skills Help: Measuring and Repairing Agent Knowledge

- 用随机掩码测量每个技能的因果效应,发现技能效果因任务而异。
- 在七种模型上验证,最高任务完成率提升47.4%,达新纪录。
- 适合需要高效推理、不改权重的智能体系统开发者使用。
LLM智能体可通过经验积累自然语言技能而无需更新权重,但当前系统将技能保留与应用决策完全依赖于LLM自身判断。我们指出,生成技能是创造性行为,适宜由判断处理;而判断技能是否真正有效则需跨任务实证证据。通过随机掩码测量每个技能的因果贡献,发现技能库存在普遍因果异质性:单个技能在某些任务中有效,另一些任务中反而有害,但总体效果相互抵消,导致全局优化方法无法察觉。为此提出ASSAY框架,分离生成与筛选:在小规模开发集上计算每个技能的因果归因,离线重构技能库,并抑制对测试任务有负效应的技能。在涵盖四种厂商的七种基础模型及两个基准(AppWorld和tau-bench)上,ASSAY持续优于现有技能筛选方法。在AppWorld最困难的划分中,DeepSeek-V3的任务目标完成率达69.3%(相对提升47.4%),成为公开方法中的新纪录;在tau-bench零售任务上,GPT-4.1相对提升8.7%,超越o4-mini、o1和GPT-4.5,且未进行任何权重修改。消融实验表明主要增益来自任务级掩码,确认瓶颈在于推理时技能与任务匹配,而非全局剔除劣质技能。代码已开源。
原文摘要 · Abstract (English)
LLM agents can improve without weight updates by accumulating natural-language skills from experience, but current systems entrust every decision about which skills to keep and how to apply them to LLM judgment alone. We argue that this conflates two distinct roles: generating a skill from experience is a creative act that judgment handles well, while deciding whether that skill actually helps requires empirical evidence across many tasks. Measuring per-skill causal contributions via randomized masking, we find that skill libraries exhibit pervasive causal heterogeneity: individual skills routinely help on some task types while hurting on others, yet their opposing effects cancel in aggregate, making them invisible to global curation methods. We propose ASSAY, a framework that separates generation from curation: it computes a per-skill causal attribution on a small development set, restructures the library offline, and suppresses skills with negative predicted effect for each test task. Across seven base models spanning four providers and two benchmarks (AppWorld and tau-bench), ASSAY consistently improves over prior skill-curation approaches. On AppWorld's hardest split, DeepSeek-V3 achieves 69.3% task-goal completion (47.4% relative improvement), a new state of the art among all published methods including weight-tuned approaches. On tau-bench retail, GPT-4.1 improves by 8.7% relative, advancing past o4-mini, o1, and GPT-4.5 on the public leaderboard without any weight modification. Ablation traces the dominant gain to per-task masking, confirming that the bottleneck is matching skills to tasks at inference time, not removing bad skills globally. Code is available at https://github.com/aiming-lab/assay.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。