arXiv:2511.21686cs.CLcs.AI2025-11被引 3

去中心化多智能体数据生成框架,提升效率与灵活性。

Matrix: Peer-to-Peer Multi-Agent Synthetic Data Generation Framework

  • 用分布式消息队列实现无中心协调的智能体协作
  • 在相同硬件下生成速度提升2-15倍,质量不降
  • 适合需要高并发、可扩展的数据生成场景

合成数据在训练大语言模型中日益重要,尤其当真实数据稀缺、昂贵或涉及隐私时。许多生成任务需多智能体协同,以产出更高质量、更多样、结构更丰富的数据。然而,现有框架多依赖中心化编排器,存在扩展瓶颈,或仅适配特定领域,灵活性差。我们提出 extbf{Matrix},一种去中心化框架,将控制流与数据流均表示为通过分布式队列传递的序列化消息。该对等设计消除了中心协调器。每个任务由轻量级智能体独立推进,而计算密集型操作(如LLM推理或容器化环境)由分布式服务处理。基于Ray构建,Matrix可支持数万个并发智能体工作流,具备模块化、可配置特性,便于适配多种数据生成场景。我们在多智能体协作对话、基于网页的推理数据提取、客户服务中的工具使用轨迹生成等多样场景中评估Matrix。所有情况下,其在相同硬件资源下实现2–15倍更高的数据生成吞吐量,且输出质量未下降。

原文摘要 · Abstract (English)

Synthetic data has become increasingly important for training large language models, especially when real data is scarce, expensive, or privacy-sensitive. Many such generation tasks require coordinated multi-agent workflows, where specialized agents collaborate to produce data that is higher quality, more diverse, and structurally richer. However, existing frameworks for multi-agent synthesis often depend on a centralized orchestrator, creating scalability bottlenecks, or are hardcoded for specific domains, limiting flexibility. We present \textbf{Matrix}, a decentralized framework that represents both control and data flow as serialized messages passed through distributed queues. This peer-to-peer design eliminates the central orchestrator. Each task progresses independently through lightweight agents, while compute-intensive operations, such as LLM inference or containerized environments, are handled by distributed services. Built on Ray, Matrix scales to tens of thousands of concurrent agentic workflows and provides a modular, configurable design that enables easy adaptation to a wide range of data generation workflows. We evaluate Matrix across diverse synthesis scenarios, such as multi-agent collaborative dialogue, web-based reasoning data extraction, and tool-use trajectory generation in customer service environments. In all cases, Matrix achieves $2$--$15\times$ higher data generation throughput under identical hardware resources, without compromising output quality.

多智能体合成数据去中心化高效生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。