让大模型流式应用像流水线一样高效运行，自动管理排队和资源。
AiFlow: Token-Native Reactive Orchestration with Bounded Backpressure for Streaming LLM Applications
- 用令牌级反应式架构统一处理不同服务的差异，通过上下文事件流驱动流程。
- 相比传统方式，应用启动延迟降低70.9%~94.7%，队列深度减少93.7%~96.5%。
- 适合需要低延迟、高可控性的实时大模型应用开发者使用。
大语言模型应用正越来越多地以流式工作流形式运行，涵盖检索、工具调用、安全过滤和多智能体协作。现有框架虽暴露了服务端差异，但工作流节点仍把生成当作粗粒度请求-响应步骤，队列管理、资源分配、排序和背压处理依赖临时回调代码。本文提出AiFlow，一种以令牌为单位的反应式编排模型，将服务差异归一化为带类型的Context<T>事件，沿有向流图传播。每个节点由节点守护者管理，声明并强制执行本地队列上限、并发数、排序策略、溢出处理、取消传播和重试规则。我们形式化了有界内存特性，提供从紧凑DSL和JSON图结构的编译方案，并实现类型安全、状态并发和注入兼容性的静态验证。受控微基准测试、DeepSeek trace回放（30次）、在线演示、LangGraph基线、流式RAG任务及Ollama本地后端测试表明，AiFlow不改变服务端模型TTFT，但将应用TTFPT降低70.9%~94.7%，并将运行时队列深度保持在声明范围内（相比无界策略，最大队列降低93.7%~96.5%）。补充材料包含脚本、原始追踪数据、可机器读取表格、校验值和无API烟雾测试；公开实现可通过FIT框架仓库获取。
原文摘要 · Abstract (English)
Large language model (LLM) applications increasingly operate as streaming workflows combining retrieval, tool calls, safety filters, and multi-agent coordination. Although contemporary frameworks expose provider deltas, workflow nodes often treat generation as coarse request-response steps, leaving queue management, worker allocation, ordering, and backpressure to ad hoc callback code. This paper presents AiFlow, a token-native reactive orchestration model that normalizes provider deltas into typed Context<T> events propagated through a directed streaming graph. Each node is managed by a Node Guardian that declares and enforces local queue bounds, worker concurrency, ordering, overflow policy, cancellation propagation, and retry discipline. We formalize the bounded-memory property, present the compilation from a compact DSL and JSON graph form, and provide static validation for type safety, state concurrency, and injection compatibility. Controlled microbenchmarks, captured DeepSeek trace replay (30 runs), descriptive online runs, LangGraph baselines, a streaming RAG workload, and an Ollama local-backend check show that AiFlow does not alter provider-side Model TTFT but reduces Application TTFPT by 70.9-94.7\% versus aggregation and keeps runtime-owned queue depth within declared bounds (93.7-96.5\% MaxQ reduction versus unbounded policies). The supplementary artifact contains scripts, raw traces, machine-readable tables, checksums, and an API-free smoke test; the public implementation is available through the FIT Framework repository.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。