用结构化知识提升大模型执行电信运维任务的可靠性
SKILLS: Structured Knowledge Injection for LLM-Driven Telecommunications Operations
- 引入SKILLS框架，通过结构化知识文档指导大模型使用电信API
- 加了知识后，所有模型表现提升，最高达+18.9个百分点
- 适合想用大模型做电信自动化的企业和研究者
随着电信运营商加速推进AI自动化，一个关键问题仍未解决：通用大语言模型（LLM）代理能否通过真实API接口可靠执行电信运维流程，还是必须依赖结构化领域指导？我们提出SKILLS（面向LLM驱动的服务生命周期运维的结构化知识注入）框架，包含37个覆盖8个TM论坛开放API领域（TMF620、TMF621、TMF622、TMF628、TMF629、TMF637、TMF639、TMF724）的电信运维场景。每个场景基于模拟生产数据的实时假服务器、MCP工具接口和确定性评估标准（包含响应内容检查、工具调用验证和数据库状态断言）。我们在5种开源模型上对比两种条件：基础版（仅具工具访问权限）与技能增强版（集成可移植的SKILL.md文档，包含流程逻辑、API模式和业务规则）。185次测试显示，所有模型在加入技能后均有显著提升。MiniMax M2.5表现最佳（技能增强后达81.1%，提升13.5个百分点），其次为Nemotron 120B（78.4%，+18.9pp）、GLM-5 Turbo（78.4%，+5.4pp）和Seed 2.0 Lite（75.7%，+18.9pp）。
原文摘要 · Abstract (English)
As telecommunications operators accelerate adoption of AI-enabled automation, a practical question remains unresolved: can general-purpose large language model (LLM) agents reliably execute telecom operations workflows through real API interfaces, or do they require structured domain guidance? We introduce SKILLS (Structured Knowledge Injection for LLM-driven Service Lifecycle operations), a benchmark framework comprising 37 telecom operations scenarios spanning 8 TM Forum Open API domains (TMF620, TMF621, TMF622, TMF628, TMF629, TMF637, TMF639, TMF724). Each scenario is grounded in live mock API servers with seeded production-representative data, MCP tool interfaces, and deterministic evaluation rubrics combining response content checks, tool-call verification, and database state assertions. We evaluate open-weight models under two conditions: baseline (generic agent with tool access but no domain guidance) and with-skill (agent augmented with a portable SKILL.md document encoding workflow logic, API patterns, and business rules). Results across 5 open-weight model conditions and 185 scenario-runs show consistent skill lift across all models. MiniMax M2.5 leads (81.1% with-skill, +13.5pp), followed by Nemotron 120B (78.4%, +18.9pp), GLM-5 Turbo (78.4%, +5.4pp), and Seed 2.0 Lite (75.7%, +18.9pp).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。