用智能代理自动优化多模态实时应用的部署,显著降低延迟、提升吞吐。
FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications

- 通过多轮编程链引导编码代理,将简单代码转换为高效多卡部署
- 在NVIDIA B200上实现最高70倍延迟降低,2.8倍吞吐提升
- 适用于视频生成、多模态大模型等场景,尤其适合硬件生态较弱平台
实时多模态应用(如语音助手和交互式视频生成)需将异构模型组合成流水线,其高效部署依赖于对模型位置、数据流和模型内并行的特定应用决策。现有服务系统与自动并行编译器仅支持有限变换和固定负载假设,新应用的高性能部署通常需手动优化。本文提出FlashRT,一个由智能代理驱动的部署引导框架,可将开发者编写的简单参考实现,自动优化为跨多GPU的高效部署方案,灵活权衡延迟、吞吐等指标。FlashRT采用新型编程链范式,引导通用编码代理经过多轮转换:先将参考代码转化为中间表示(IR),以捕捉数据依赖与状态作用域;再通过顺序解释器验证IR,并进行静态分析识别候选优化;最后在测量驱动的优化循环中迭代实施、验证与基准测试,生成适配不同硬件预算的部署方案。在多种应用(包括视频世界模型与多模态LLM)上,FlashRT成功将参考实现转为高效部署,在NVIDIA B200 GPU上实现约70倍延迟降低与2.8倍吞吐提升;在AMD MI355X GPU上,延迟降低达峰值,吞吐提升增至3.6倍,表明代理驱动优化在成熟度较低的平台上更具扩展性。例如,对于Qwen3-Omni文本转音频推理任务,相比专家级vLLM-Omni实现,FlashRT在AMD MI355X上将响应延迟降低65%。
原文摘要 · Abstract (English)
Real-time multimodal applications, including voice agents and interactive video generation, compose heterogeneous models into pipelines whose efficient deployment requires application-specific decisions about placement, streaming, and intra-model parallelism. Existing serving systems and auto-parallelism compilers commit to limited transformations and fixed workload assumptions, so achieving high performance on a new application requires hand-crafting an efficient implementation. We present FlashRT, an agent harness that guides coding agents to lift simple developer-written reference implementations into optimized multi-GPU deployments that flexibly weigh target metrics like latency and throughput. Using a new chain-of-program paradigm, FlashRT directs a generic coding agent through a multi-pass transformation process where an agent transforms the reference into an intermediate representation (IR) to capture data dependencies and persistent-state scopes, validates this IR via a sequential interpreter, and performs static analyses to identify candidate transformations. Then, the agent iteratively implements, verifies, and benchmarks each candidate under a measurement-gated optimization loop to produce effective deployments that span different hardware budgets. Across various applications, including video world models and multimodal LLMs, FlashRT converts reference implementations into highly efficient deployments, delivering up to ~70x latency reduction and 2.8x throughput improvement on NVIDIA B200 GPUs. On AMD MI355X GPUs, FlashRT matches the peak latency reduction while increasing peak throughput improvement to 3.6x, demonstrating that agent-driven optimization can be more scalable on platforms with less mature expert optimization. In fact, for Qwen3-Omni text-to-audio inference, FlashRT reduces response latency by 65% compared to the expert vLLM-Omni implementation on AMD MI355X.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。