用分步生成提升企业自动化工作流成功率，让中小模型也能胜任生产级任务。
Evaluating LLM Trade-offs for Enterprise Automation: Lessons from Workflow Generation in a Production Enterprise Platform
- 将工作流生成拆解为变量搭建、基础模块组装和嵌套模块生成三步，提升结构正确率。
- 分步架构使最小模型成功率达95.7%，成本低至0.01美元/工作流，低于顶级模型19倍。
- 适合需低成本、高可靠性的云工程自动化团队，尤其关注结构正确性而非语义意图。
企业合规管理需快速响应不断变化的监管框架（如DORA、AI RMF、FedRAMP）并满足严格的修复时效要求。传统静态编排器在混合云环境中常失效，因事件驱动评估要求自动化代码在秒级内适配运行时上下文。本文基于六种大语言模型在生产级企业平台中生成工作流的实践经验，评估了29个真实IT自动化场景、两种生成流水线架构及每配置8次独立运行（共2,784次）。初始单体流水线结构成功率仅31.5%-82.8%（JSON schema有效性与正确UI渲染），多数模型在复杂JSON生成上表现不佳。我们设计分步式流水线，将工作流构建分解为变量骨架、基础模块组装与嵌套模块生成，将结构成功率提升至74.1%-97.8%。分析显示，成本为0.008-0.20美元/工作流，延迟低于50秒，满足交互使用需求。分步策略使较小模型（如mistral-small，95.7%成功率，0.01美元/工作流）具备生产可行性，不再依赖昂贵前沿模型。尽管mistral-medium-2505与gpt-oss-120b分别达96.1%和97.8%最高成功率，但前者成本是mistral-small的19倍。部署经验强调应区分结构正确性与语义正确性（用户意图逻辑实现），提供模型无关、可扩展的云工程自动化方案。
原文摘要 · Abstract (English)
Enterprise compliance management requires rapid adaptation to evolving regulatory frameworks (e.g., DORA, AI RMF, FedRAMP) and tight remediation SLAs. Traditional static orchestrators often fail in hybrid cloud environments where event-driven assessments demand that automation code adapt to runtime context in seconds. This paper presents lessons learned from evaluating six large language models for AI-driven workflow generation in a production enterprise platform, benchmarked across 29 real-world IT automation scenarios, two generation pipeline architectures, and eight independent runs per prompt-model-pipeline configuration (2,784 runs total). Our initial pipeline used monolithic workflow generation, achieving 31.5-82.8% structural success rates (JSON schema validity and correct UI rendering), with most models struggling on complex JSON generation. We developed a redesigned piecewise pipeline that decomposes workflow construction into variable scaffolding, base block assembly, and nested block generation, raising structural success to 74.1-97.8% across all models. We analyze production tradeoffs including cost (USD 0.008-0.20 per workflow), latency (under 50s for interactive use), and model selection. Piecewise decomposition enables smaller models (e.g., mistral-small at 95.7% structural success and USD 0.01 per workflow) to reach production viability, removing dependency on expensive frontier models. While mistral-medium-2505 and gpt-oss-120b achieved the highest structural success (96.1% and 97.8%), mistral-medium-2505 carries a 19x cost premium versus mistral-small. Our deployment lessons highlight the need to separate structural validity from semantic correctness (logical fulfillment of user intent) and provide a solution for model-agnostic, scalable automation in cloud engineering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。