让多个小模型协作训练，不用中心控制器也能稳步提升性能。
SAT: Sequential Agent Tuning for Coordinator Free Plug and Play Multi-LLM Training with Monotonic Improvement Guarantees
- 按顺序更新每个小模型，用动态策略估计优势，避免团队策略漂移。
- 三组40亿参数模型总和超越320亿参数大模型，性能领先3.9%。
- 任意换更强模型都不用重训其他成员，理论保证性能只会变好。
参数量庞大的大语言模型虽表现优异，但部署成本高昂。近期研究尝试使用多个小型高效模型协同工作，以达到甚至超越单一大型模型的性能。然而，联合更新多个智能体会导致分布漂移累积，使训练过程难以协调与稳定。本文提出一种无需协调器的序列化智能体调优（SAT）方法：将团队视为可分解策略，采用块坐标更新方式对各智能体进行独立训练。具体地，设计了一种依赖于动态团队策略的序列感知在线优势估计器，并为每个智能体引入基于KL散度的信任区域，以隔离状态访问频率的漂移。理论上，该框架提供两项关键保障：一是保证训练过程单调提升，增强稳定性；二是建立可插拔不变性：任意智能体升级为更强模型后，无需重训其余成员，且整体性能上限得到形式化提升。实验上，由三个40亿参数智能体组成的团队（总计120亿参数）在AIME24/25基准测试中平均比更庞大的Qwen3-32B高出3.9%。通过替换为两个80亿参数模型，复合得分进一步提升10.4%。代码与证明附录已开源于https://github.com/Yydc/SAT-AAMAS。
原文摘要 · Abstract (English)
Large language models (LLMs) with a large number of parameters achieve strong performance but are often prohibitively expensive to deploy. Recent work explores using teams of smaller, more efficient LLMs that collectively match or even outperform a single large model. However, jointly updating multiple agents introduces compounding distribution shifts, making coordination and stability during training difficult. We address this by introducing Sequential Agent Tuning (SAT), a coordinator-free training paradigm. SAT represents the team as a factorized policy and employs block-coordinate updates over agents, enabling scalable, decentralized training without a central controller. Specifically, we develop a sequence-aware, on-policy advantage estimator that conditions on the evolving team policy, coupled with per-agent KL trust regions that isolate occupancy drift. Theoretically, this framework provides two critical guarantees. First, it ensures monotonic improvement, stabilizing the training process. Second, it establishes provable plug-and-play invariance: any agent can be upgraded to a stronger model without retraining the rest of the team, with a formal guarantee that the performance bound improves. Empirically, a team of three 4B agents (12B total) trained with SAT surpasses the much larger Qwen3-32B on AIME24/25 benchmarks by 3.9\% on average. We validate our plug-and-play theory by swapping in two 8B agents, which boosts the composite score by 10.4\%. We provide code and appendix of proof at https://github.com/Yydc/SAT-AAMAS
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。