用影响得分优化树搜索，提升多智能体系统训练效率
Efficient Multi-Agent System Training with Data Influence-Oriented Tree Search
- 引入影响得分替代传统Q值，更精准筛选对训练有帮助的数据
- 在8个数据集上验证，显著提升模型性能且降低计算开销
- 适合关注大模型多智能体训练效率的研究者与工程师
基于蒙特卡洛树搜索（MCTS）的方法为增强基于大语言模型的多智能体系统（MAS）自训练提供了可行路径，通过Q值估计单个智能体的贡献。然而，仅依赖Q值识别有效数据可能偏离数据合成目标，因重点应是选择最能提升模型训练的数据。为此，我们提出数据影响导向的树搜索（DITS），融合影响得分指导树搜索与数据选择。通过影响得分，可有效识别对系统改进最具影响力的样本，从而提升模型表现。此外，我们设计了针对非可微指标的影响得分估算方法，显著减少计算开销，仅利用推理过程完成。在八个多智能体数据集上的大量实验表明，该方法具备鲁棒性与有效性。值得注意的是，将更多推理资源用于估算影响得分而非Q值，能更高效地增强模型训练。
原文摘要 · Abstract (English)
Monte Carlo Tree Search (MCTS) based methods provide promising approaches for generating synthetic data to enhance the self-training of Large Language Model (LLM) based multi-agent systems (MAS). These methods leverage Q-values to estimate individual agent contributions. However, relying solely on Q-values to identify informative data may misalign with the data synthesis objective, as the focus should be on selecting data that best enhances model training. To address this discrepancy, we propose Data Influence-oriented Tree Search (DITS), a novel framework that incorporates influence scores to guide both tree search and data selection. By leveraging influence scores, we effectively identify the most impactful data for system improvement, thereby enhancing model performance. Furthermore, we derive influence score estimation methods tailored for non-differentiable metrics, significantly reducing computational overhead by utilizing inference computations. Extensive experiments on eight multi-agent datasets demonstrate the robustness and effectiveness of the proposed methods. Notably, our findings reveal that allocating more inference resources to estimate influence scores, rather than Q-values, during data synthesis can more effectively and efficiently enhance model training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。