用一次大模型重写就能优化技能描述，大幅降低人工成本。
A Single Rewrite Suffices: Empirical Lessons from Production Skill Description Optimization
- 用大模型重写有歧义的技能描述，解决路由错误问题。
- 自动优化后平均F1达79.2%，接近人工调优的79.4%。
- 只需一次重写即可实现主要提升，适合大规模生产系统。
企业级AI代理通过匹配用户查询与自然语言技能描述来分配任务。当多个技能描述重叠时，路由大模型会产生误判，称为技能碰撞。随着技能数量增至数十个，手动调优描述以保证路由准确率成为重大工程瓶颈。我们在一个包含9个技能、372个回归案例的生产级企业聊天代理上部署了自动化描述优化流水线。该流水线生成的描述平均F1达到79.2%，与人工调优的79.4%几乎持平（每技能平均差异-0.20%，低于0.78%的多种子噪声水平），同时将每技能工程耗时从120分钟降至3.8分钟（提速32倍）。进一步在生产系统和ToolBench（16k工具）上进行系统性消融实验发现，仅使用任意正负样本进行一次大模型重写，即可捕获大部分改进效果。其他设计选择（迭代预算、反馈信号构成、混淆对双编辑、训练集大小）对最终F1的影响均小于0.5%。描述优化可缓解因描述重叠导致的技能碰撞，但无法解决两技能本意范围确实重叠的情况。我们提出一种诊断方法（训练集与验证集F1差距大），用于识别此类需架构层面干预的问题。
原文摘要 · Abstract (English)
Enterprise AI agents route user queries to specialized skills by matching queries against natural language skill descriptions. When two skills share overlapping descriptions, the routing LLM misroutes queries, a failure we term skill collision. As agents scale to dozens of skills, manually tuning descriptions to maintain routing accuracy becomes a significant engineering bottleneck. We deploy an automated description optimization pipeline on a production enterprise group chat agent (9 skills, 372 regression cases). The pipeline produces descriptions averaging 79.2% F1, matching manually tuned descriptions at 79.4% F1 (average per-skill difference -0.20%, within the 0.78% multi-seed noise floor), while reducing per-skill engineering effort from 120 minutes to 3.8 minutes (32 times speedup). We then examine which pipeline components actually drive this match. Systematic ablation on both the production system and ToolBench (16k tools) reveals that a single LLM rewrite using any available false-positive and false-negative cases captures most of the available improvement. Other design choices we tested (iteration budget, feedback signal composition, dual editing of confused pairs, and training set size) each affect final F1 by less than 0.5%. Description optimization addresses skill collisions caused by overlapping descriptions but cannot resolve cases where two skills intended scopes genuinely overlap. We identify a diagnostic (a large train-validation F1 gap) that flags the latter cases for architectural rather than text-level intervention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。