让大模型工作流在不同运行环境间无缝切换，降低成本。
Substrate-Portable Execution for Production LLM Workflows
- 工作流定义与执行环境解耦，一套代码适配实时、异步、批量三种模式。
- 批量执行成本降低50%，与实时服务并行运行无质量损失。
- 适合需要多场景部署的大模型应用开发者，尤其关注成本优化者。
生产级大模型代理需在实时服务、异步任务和高吞量批处理等多种模式下运行工具调用循环、检索链及组合式工作流，但执行语义常绑定于特定运行时。我们在 Rufus——一个服务于数百万亚马逊用户的对话式AI助手——中遇到此可移植性问题，其拥有庞大的工具库。Rufus支持实时响应、异步后台任务及评估与内容预生成等高吞量批处理任务。每种模式有不同服务目标，通常使用独立运行时。复用流式编排会导致异步和批处理任务阻塞，并无法使用提供50%折扣的批量推理API。我们提出一种绑定自适应的代理执行平台，将工作流定义与执行底座分离。开发者仅需定义一次类型化数据流图，平台即可将其编译为：本地流式处理（实时）、持久化的AWS SWF编排（异步）或分布式Apache Flink流处理（批处理）。无需修改工作流代码。大模型推理以可暂停的图节点形式表示，其行为随底座变化：在线流式交付、异步持久重试、离线批量提交。我们在五种编排模式下验证了数十个生产级代理配置：单次推理RAG、迭代ReAct、组合式PreAct、条件路由与多代理深度研究。三类绑定均未发现输出质量差异。批量执行成本符合公开批量API定价，同时在生产规模下与流式路径并行运行。
原文摘要 · Abstract (English)
Production LLM agents execute tool-calling loops, retrieval chains, and compositional workflows in multiple modes, yet execution semantics are often coupled to one runtime. We encountered this portability problem in Rufus, a conversational AI assistant with a large tool catalog that serves millions of Amazon customers. Rufus supports real-time serving, asynchronous background tasks, and high-volume batch workloads such as evaluation and content pregeneration. Each mode has distinct service-level objectives and typically uses a separate runtime. Reusing streaming orchestration makes asynchronous and batch workloads blocking and prevents use of batch inference APIs, which offer a 50 percent discount at published prices. We present a binding-adaptive agent execution platform that separates workflow definition from execution substrate. Developers define a workflow once as a typed dataflow graph. The platform compiles the graph to in-process streaming for real-time serving, durable AWS SWF orchestration for asynchronous execution, or distributed Apache Flink stream processing for batch inference. No workflow code changes are required. LLM inference is represented as a suspendable graph node whose behavior depends on the substrate: streaming delivery online, durable retry asynchronously, and batched submission offline. We validated dozens of production agent configurations across five orchestration patterns: single-inference RAG, iterative ReAct, compositional PreAct, conditional routing, and multi-agent deep research. Across all three bindings, we found no detectable difference in output quality. Batch execution reduced per-query inference cost in line with published batch API pricing while operating alongside the streaming path at production scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。