评测大模型查询企业数据时的结构化语义规划能力，发现约束越多不一定越好。
SemPlan: Benchmarking Structured Semantic Planning for LLM-Based Queries over Enterprise Data
- 设计双语合成基准，用1800个案例测试不同查询架构的性能
- 结构化规划方案正确率最高，但所有方法整体准确率仍不足26%
- 复杂机制能降低无效调用和拒绝率，但难解决多轮一致性问题
面向企业数据的自然语言接口需将模糊请求转化为受控、可执行的行为，同时防范无效查询、策略失败、成本过高及不确定性。本文提出SemPlan基准，采用包含1800个案例的确定性合成双语数据集（英文与巴西葡萄牙语），其中1200个为固定科学评估子集。在相同模型配置下比较四种架构：直接生成SQL（A1）、有限工具代理基线（A2）、结构化语义请求生成+确定性规划执行（A3），以及澄清/状态化语义计划变体（A4）。在4800条主记录上，答案正确率均偏低：A1为22.25%，A2为22.58%，A3为25.67%，A4为24.25%。A3在预设配对分析中显著优于A1、A2和A4，而A1保持最高策略正确率和最低不安全或无效率。A4的平均API成本最低，虚假拒绝率也最低。在150例稳定性子集上，答案重复性介于92.00%至98.67%之间。结果表明存在权衡而非绝对优劣：额外结构约束改变了失败模式与效率，但未单调提升正确率，也无法彻底解决模糊性与多轮状态一致性问题。
原文摘要 · Abstract (English)
Natural-language interfaces to enterprise data must translate underspecified requests into governed, executable behavior while controlling invalid queries, policy failures, cost, and nondeterminism. SemPlan Benchmark evaluates this architectural design space with a deterministic synthetic bilingual benchmark containing 1,800 cases in English and Brazilian Portuguese; 1,200 cases form the frozen scientific evaluation subset. Four architectures are compared under the same model configuration: direct SQL generation (A1), a bounded tool-agent baseline (A2), structured semantic-request generation followed by deterministic planning and execution (A3), and a clarification/stateful semantic-plan variant (A4). Across 4,800 primary records, answer correctness was low in absolute terms: 22.25% for A1, 22.58% for A2, 25.67% for A3, and 24.25% for A4. A3 had the highest observed correctness and significantly exceeded A1, A2, and A4 in the pre-specified paired correctness analysis, while A1 retained the highest policy-correct rate and the lowest unsafe-or-invalid rate. A4 had the lowest mean API cost and lowest false-refusal rate. On a preselected 150-case stability subset, answer-correct repeatability ranged from 92.00% to 98.67%. The results support a trade-off interpretation rather than a universal ranking: additional structural constraints changed failure modes and efficiency, but did not monotonically improve correctness or solve ambiguity and multi-turn state consistency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。