用大模型在线学调优，让多阶段推理流水线更省资源、更快响应。
SAIR: Cost-Efficient Multi-Stage ML Pipeline Autoscaling via In-Context Reinforcement Learning
- 用大模型做在线强化学习控制器，不靠梯度更新，凭反馈历史自我改进。
- 在三种负载下，延迟降低最高50%，资源成本减少高达97%。
- 无需离线训练，能准确识别瓶颈，适合生产环境的智能调度场景。
多阶段机器学习推理流水线因资源异构、跨阶段耦合及动态瓶颈迁移而难以自动伸缩。我们提出SAIR，一个利用大模型作为上下文强化学习控制器的自动伸缩框架，通过奖励标记的交互历史在线优化策略，无需梯度更新。SAIR结合帕累托占优奖励设计与可证明分离边界，采用惊喜度引导的经验检索提升上下文效率，并通过用户空间CUDA拦截实现细粒度GPU速率控制。我们提供了后悔值分析，将误差分解为检索覆盖和大模型选择两部分。在四个真实推理流水线上，面对三种工作负载模式，SAIR在部署基线中达到最优或并列最优的P99延迟与有效资源成本，延迟最高改善50%，有效成本最高降低97%（在GPU速率控制假设下），瓶颈检测准确率达86%，且无需离线训练。
原文摘要 · Abstract (English)
Multi-stage ML inference pipelines are difficult to autoscale due to heterogeneous resources, cross-stage coupling, and dynamic bottleneck migration. We present SAIR, an autoscaling framework that uses an LLM as an in-context reinforcement learning controller, improving its policy online from reward-labeled interaction histories without gradient updates. SAIR combines Pareto-dominance reward shaping with a provable separation margin, surprisal-guided experience retrieval for context efficiency, and fine-grained GPU rate control via user-space CUDA interception. We provide regret analysis decomposing error into retrieval coverage and LLM selection components. On four ML serving pipelines under three workload patterns, SAIR achieves the best or tied-best P99 latency and effective resource cost among deployed baselines, improving P99 by up to 50% and reducing effective cost by up to 97% (under GPU rate-control assumptions), with 86% bottleneck detection accuracy and no offline training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。