通过确定性执行门控，高效筛选大模型技能库中可执行的任务。
Don't Offer What Can't Be Done: Deterministic Executability Gating for LLM Skill Selection at Scale
- 引入三阶段流程，先语义匹配再用硬停止条件过滤不可执行技能。
- 在75万条消息中移除超百万个不可执行技能对，节省90.5%上下文开销。
- 确保无效技能不干扰模型选择，适合高并发生产级智能助手部署。
面向大规模技能库的生产级大模型代理面临语义相关性无法解决的问题：某技能虽与用户话题匹配，但在当前账户状态下无法执行。本文提出部署于Wix客服助手Helpmate的三阶段选择管道：首先，基于语义匹配召回与十项技能族相关的消息，不依赖账户状态；其次，通过确定性执行门控剔除满足内部硬停止条件的候选技能。因门控与技能评估相同的退出断言，且均观测最新权威状态，在断言一致的前提下，所有被阻断的候选均无法完成执行。最后，由大模型从剩余候选中决定调用。在267,600次对话共756,600条用户消息的上线分析中，语义匹配保留174,927条消息（23.1%）；在该匹配流中，门控移除了1,039,462/1,749,270个技能-消息对（59.4%），节省2.288亿技能描述令牌——占语义匹配后技能描述总量的59.1%。整体上，语义匹配与执行门控联合使技能描述上下文减少90.5%，远低于向每条消息暴露全部十项技能的情况。为验证此剪枝是否影响模型行为而非仅上下文大小，我们重放了包含1,000次风险增强对话的样本集，全量技能暴露下模型在78次对话中选择了生产中被阻断的技能（7.8%）。反事实结果表明，确定性门控有效防止不可执行候选影响模型选择，但未宣称其对下游工具执行或客户结果的影响。
原文摘要 · Abstract (English)
Production LLM agents that select from large skill libraries face a limitation that semantic relevance alone cannot resolve: a skill may match a user's topic yet be impossible to execute in the current account state. We present a deployed three-stage selection pipeline for Helpmate, Wix's customer-care assistant. First, a recall-oriented semantic matcher identifies messages related to a ten-skill domain family without consulting account state. Second, a deterministic executability gate removes candidates whose internal hard-stop conditions hold. Because the gate and the skill evaluate the same exit predicates, every blocked candidate would be unable to complete under the same account state, provided predicate parity is preserved and both checks observe fresh authoritative state. Finally, the LLM decides whether to invoke one of the remaining candidates. In a post-launch production analysis of 756.6K user messages across 267.6K conversations, semantic matching retained 174,927 messages (23.1%). Within this matched stream, the gate removed 1,039,462 of 1,749,270 skill-message pairs (59.4%), saving 228.8 million skill-description tokens -- 59.1% of the post-semantic skill-description footprint. Together, semantic matching and executability gating reduced skill-description context by 90.5% relative to exposing all ten skills to every message. To test whether this pruning affects model behavior rather than context size alone, we replayed a risk-enriched cohort of 1,000 conversations with all ten skills exposed. The model selected a production-blocked skill in 78 conversations (7.8%). This counterfactual result shows that deterministic gating prevents non-executable candidates from influencing model selection, while not claiming downstream tool execution or customer-outcome effects.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。