评估大模型在客服场景中分解复杂查询的工具规划能力，发现模型对工具理解仍有明显短板。
Tool-Aware Planning in Contact Center AI: Evaluating LLMs through Lineage-Guided Query Decomposition
- 通过线索引导的查询分解，将业务问题拆解为可执行的结构化与非结构化工具步骤。
- 14个大模型测试显示，最佳综合得分84.8%（Claude-3-7-Sonnet），但超4步计划成功率骤降。
- 提出可复现的评估框架与高质量计划生成流程，适合研究智能客服与工具调用的开发者。
我们提出一个面向客服场景的领域驱动框架与基准，用于评估大模型在生成工具感知计划方面的能力。目标是回答业务洞察类查询，需将查询分解为可在结构化工具（Text2SQL/Snowflake）和非结构化工具（RAG/通话记录）上执行的步骤，并显式标注依赖关系以支持并行处理。贡献包括：(i) 一种基于参考的计划评估框架，包含七维度指标（如工具-提示对齐、查询遵循度）的度量评估器与单次匹配评估器；(ii) 通过评估器→优化器循环迭代精炼计划的數據構建方法，生成高质量的计划线索（有序的计划修订序列），降低人工成本；(iii) 对14个不同规模和家族的大模型进行大规模研究，评估其在有无计划线索条件下，将查询分解为可执行、分步、工具分配的计划的能力。实证表明，大模型在复合查询及超过4步的计划中表现不佳（通常5-15步）；最佳综合评分达84.8%（Claude-3-7-Sonnet），而最强单次匹配率在“A+”等级仅49.75%（o3-mini）。计划线索整体带来混合收益，但对部分顶尖模型有提升，且显著提高多数计划的可执行性。结果凸显了在工具理解上的持续差距，尤其在工具-提示对齐与工具使用完整性方面；更短、更简单的计划显著更容易完成。该框架与发现为评估和改进客服场景中工具调用的智能体规划提供了可复现路径。
原文摘要 · Abstract (English)
We present a domain-grounded framework and benchmark for tool-aware plan generation in contact centers, where answering a query for business insights, our target use case, requires decomposing it into executable steps over structured tools (Text2SQL (T2S)/Snowflake) and unstructured tools (RAG/transcripts) with explicit depends_on for parallelism. Our contributions are threefold: (i) a reference-based plan evaluation framework operating in two modes - a metric-wise evaluator spanning seven dimensions (e.g., tool-prompt alignment, query adherence) and a one-shot evaluator; (ii) a data curation methodology that iteratively refines plans via an evaluator->optimizer loop to produce high-quality plan lineages (ordered plan revisions) while reducing manual effort; and (iii) a large-scale study of 14 LLMs across sizes and families for their ability to decompose queries into step-by-step, executable, and tool-assigned plans, evaluated under prompts with and without lineage. Empirically, LLMs struggle on compound queries and on plans exceeding 4 steps (typically 5-15); the best total metric score reaches 84.8% (Claude-3-7-Sonnet), while the strongest one-shot match rate at the "A+" tier (Extremely Good, Very Good) is only 49.75% (o3-mini). Plan lineage yields mixed gains overall but benefits several top models and improves step executability for many. Our results highlight persistent gaps in tool-understanding, especially in tool-prompt alignment and tool-usage completeness, and show that shorter, simpler plans are markedly easier. The framework and findings provide a reproducible path for assessing and improving agentic planning with tools for answering data-analysis queries in contact-center settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。