用大模型把专家知识转成可执行的逻辑规则,提升系统安全可靠性。
Integrating Expert Knowledge into Logical Programs via LLMs
- 通过代码执行反馈循环,让大模型自动修正逻辑错误。
- 950个工程提示下,多数模型语法正确但逻辑实现能力差异大。
- 适合关注工业系统安全与自纠错模型选型的研究者。
本文提出ExKLoP框架,评估大语言模型(LLMs)将专家知识融入逻辑推理系统的能力。该能力在工程领域尤为重要,例如将制造商推荐的操作范围直接嵌入自动化监控系统。通过模拟专家验证步骤,如范围检查和约束验证,确保系统安全与可靠性。本方法系统评估大模型生成的逻辑规则,涵盖语法流畅性与逻辑正确性。我们还探索了基于代码执行结果的迭代反馈机制对模型自我修正能力的影响。ExKLoP提供一个可扩展数据集,包含130个工程前提、950个提示及对应验证点,支持全面基准测试,并可控制任务复杂度与实验规模。利用合成数据生成方法,在Llama3、Gemma3、Codestral和QwenCoder等多款LLM上进行广泛实证评估。结果显示,多数模型生成的代码语法近乎完美,且能有效将专家知识转化为正确代码;然而,其逻辑规则实现能力存在显著差异,自我改进能力也参差不齐。总体而言,ExKLoP为筛选适用于自纠正系统的有效模型提供了可靠平台,并清晰揭示了各类错误类型。
原文摘要 · Abstract (English)
This paper introduces ExKLoP, a novel framework designed to evaluate how effectively Large Language Models (LLMs) integrate expert knowledge into logical reasoning systems. This capability is especially valuable in engineering, where expert knowledge-such as manufacturer-recommended operational ranges-can be directly embedded into automated monitoring systems. By mirroring expert verification steps, tasks like range checking and constraint validation help ensure system safety and reliability. Our approach systematically evaluates LLM-generated logical rules, assessing both syntactic fluency and logical correctness in these critical validation tasks. We also explore the models' capacity for self-correction via an iterative feedback loop based on code execution outcomes. ExKLoP presents an extensible dataset comprising 130 engineering premises, 950 prompts, and corresponding validation points. It enables comprehensive benchmarking while allowing control over task complexity and scalability of experiments. We leverage the synthetic data creation methodology to conduct extensive empirical evaluation on a diverse set of LLMs including Llama3, Gemma3, Codestral and QwenCoder. The results reveal that most models generate nearly perfect syntactically correct code and exhibit strong performance in translating expert knowledge into correct code. At the same time, while most LLMs produce nearly flawless syntactic output, their ability to correctly implement logical rules varies, as does their capacity for self-improvement. Overall, ExKLoP serves as a robust evaluation platform that streamlines the selection of effective models for self-correcting systems while clearly delineating the types of errors encountered.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。