让大模型提前生成可执行代码,实现无依赖的稳定运行。
Compiled AI: Deterministic Code Generation for LLM-Based Workflow Automation
- 用限定模板生成代码,确保输出可预测、可审计。
- 函数调用任务完成率达96%,1000次调用节省57倍令牌开销。
- 适合医疗等高可靠性要求场景,兼顾安全与成本控制。
我们研究编译式AI,即大语言模型在编译阶段生成可执行代码,后续工作流无需再次调用模型即可确定性执行。该范式借鉴了声明式管道优化(DSPy)和混合神经符号规划(LLM+P)的思路,本文重点探讨其在高风险企业工作流中的应用,尤其在医疗领域对可靠性和可审计性的需求。通过将生成限制在预验证模板内的业务逻辑函数,编译式AI以牺牲运行时灵活性为代价,获得更高的确定性、可审计性、成本效益和更低的安全风险。我们提出:(i) 一种受限的LLM代码生成系统架构;(ii) 四阶段生成-验证流水线,将概率输出转化为生产级代码;(iii) 评估框架,涵盖令牌摊销、确定性、可靠性、安全性和成本等指标。在两类任务上评估:函数调用(BFCL,n=400)和文档智能(DocILE,n=5,680发票)。在函数调用任务中,编译式AI实现96%任务完成率且零执行令牌,约17次交易后即可回本,1000次交易时令牌消耗减少57倍。在文档智能任务中,我们的Code Factory变体在关键字段提取(KILE: 80.0%)上达到与直接调用模型相当的性能,同时实现最高行项目识别准确率(LIR: 80.4%)。对135个测试案例的安全评估显示,提示注入检测准确率达96.7%,静态代码安全分析准确率为87.5%,且无误报。
原文摘要 · Abstract (English)
We study compiled AI, a paradigm in which large language models generate executable code artifacts during a compilation phase, after which workflows execute deterministically without further model invocation. This paradigm has antecedents in prior work on declarative pipeline optimization (DSPy) and hybrid neural-symbolic planning (LLM+P); our contribution is a systems-oriented study of its application to high-stakes enterprise workflows, with particular emphasis on healthcare settings where reliability and auditability are critical. By constraining generation to narrow business-logic functions embedded in validated templates, compiled AI trades runtime flexibility for predictability, auditability, cost efficiency, and reduced security exposure. We introduce (i) a system architecture for constrained LLM-based code generation, (ii) a four-stage generation-and-validation pipeline that converts probabilistic model output into production-ready code artifacts, and (iii) an evaluation framework measuring operational metrics including token amortization, determinism, reliability, security, and cost. We evaluate on two task types: function-calling (BFCL, n=400) and document intelligence (DocILE, n=5,680 invoices). On function-calling, compiled AI achieves 96% task completion with zero execution tokens, breaking even with runtime inference at approximately 17 transactions and reducing token consumption by 57x at 1,000 transactions. On document intelligence, our Code Factory variant matches Direct LLM on key field extraction (KILE: 80.0%) while achieving the highest line item recognition accuracy (LIR: 80.4%). Security evaluation across 135 test cases demonstrates 96.7% accuracy on prompt injection detection and 87.5% on static code safety analysis with zero false positives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。