解决大模型生成电力系统代码时因接口知识不足导致的错误问题。
Knowledge Boundary Probing and Demand-Guided Intervention for LLM-Based Power System Code Generation
- 通过自然语言查询与真实代码匹配，构建可执行验证的基准测试集。
- 发现大模型在版本化仿真库中常误用函数名和参数，准确率提升32至56点。
- 无需微调或云端推理，适合需本地部署的电力系统研究机构使用。
大型语言模型（LLMs）正被用于自动化电力系统分析，但许多电网公司和科研实验室因保密、合规、可复现性及成本原因，需本地部署。这使开源权重模型的可靠性成为部署难题。我们发现，电力系统代码生成中的首次失败主要并非源于推理能力不足，而是由结构化的接口知识边界错误导致：如虚构函数名、误用参数、错误处理版本化仿真库结果表。为此，我们提出PowerCodeBench，一个执行验证的基准生成器，将自然语言操作员查询与pandapower代码及数值真值配对；设计L0-L3文档驱动探查流程，测量各模型的接口知识谱；并引入边界感知干预机制，结合查询端的接口需求估计、针对性的主动文档注入与路由式反应修正。在2,000任务的冻结版本上，评估了十款开源大模型（1.5B–480B参数）和四款商用中端API。该干预使所有7B以上参数的开源模型及所有商用API准确率提升32至56个百分点。70B–120B范围的开源模型达到商用中端水平，而Llama-3.1-405B与Qwen3-Coder-480B表现领先。目标提示在保持全上下文准确率上限的同时，仅消耗41%的提示词成本。结果表明，无需微调或云推理，即可实现可靠本地化大模型辅助电网分析工作流。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly used to automate power-system analysis, but many utilities and energy-research labs require on-premise serving for confidentiality, regulatory, reproducibility, and cost reasons. This makes the reliability of open-weight models a deployment issue. We show that first-pass failures in power-system code generation are dominated not by reasoning alone, but by structured API-knowledge boundary errors: hallucinated function names, misused parameters, and mishandled result tables in versioned simulation libraries. We introduce PowerCodeBench, an execution-validated benchmark generator that pairs natural-language operator queries with pandapower code and numerical ground truth; an L0-L3 documentation-driven probing procedure that measures per-model API knowledge profiles; and a boundary-aware intervention that combines query-side API demand estimation with targeted proactive documentation injection and routed reactive correction. On a 2,000-task frozen release, we evaluate ten open-weight LLMs (1.5B-480B parameters) and four commercial mid-tier APIs. The intervention improves every evaluated open-weight model of at least 7B parameters and every commercial API by 32 to 56 accuracy points. Open-weight models in the 70B-120B range match the commercial mid-tier accuracy range, while Llama-3.1-405B and Qwen3-Coder-480B lead the panel. The targeted prompts preserve the full-context accuracy ceiling while using 41% of the prompt-token cost. The result is an accuracy-side, deployment-time path toward reliable on-premise LLM assistance for grid-analysis workflows without fine-tuning or cloud inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。