为多智能体自动研究系统设计统一描述框架,方便比较与改进。
A Vocabulary for Multi-Agent Automated Research Systems
- 定义8类核心要素,涵盖角色、操作、通信与决策机制
- 同一任务多次运行产生轨迹分布,而非单一结果
- 区分生成味觉与评估味觉,帮助诊断系统缺陷
我们提出一套用于构建单或多智能体自动研究系统的术语体系,以简化设计选择的描述与比较。该词汇表明确指定:1)智能体身份,2)系统可用操作,3)操作调用权限,4)智能体间通信方式,5)运行内及跨运行信息可见性,6)下一步行动选择逻辑,7)运行起始方式,8)输出评估方法。一次运行轨迹记录从输入任务到产出成果的全过程。由于智能体、操作和初始化可能具有随机性,相同任务的多次运行会形成轨迹分布而非唯一行为。该词汇表将结构设计问题(如何时通信、增减能力或跨运行传递信息)转化为可验证的选择。同时,将评估器作为系统组成部分,因报告收益依赖代理评分与真实质量的匹配程度。这种分离也将模糊的‘缺乏品味’批评拆分为两类可分别解决的问题:生成味觉指未观测评分前系统提出新颖轨迹的速率;评估味觉指代理评分与应匹配质量之间的差距。我们在近期自研究系统中实例化该词汇表,证明其能覆盖结构差异显著的设计。
原文摘要 · Abstract (English)
We introduce a vocabulary for automated research systems built from one or more agents to make their design choices easier to describe and compare. The vocabulary specifies 1) who the agents are, 2) what operations are available in the system, 3) who may invoke them, 4) how agents communicate, 5) what information is visible within and across runs, 6) how the next action is chosen, 7) how a run begins, and 8) how outputs are evaluated. A trajectory records one run from the input task to the returned artifact. Because agents, operations, and initialization may be stochastic, repeated runs on the same task induce a distribution over trajectories rather than a single behavior. Our vocabulary turns structural design questions, such as when agents should communicate, gain or lose a capability, or carry information across runs, into testable choices. It also makes the evaluator a component of the system, since reported gains depend on how closely the proxy score matches true quality. That separation also splits the vague complaint that these systems lack taste into two failures with different solutions. Generative taste is the rate at which a system proposes novel trajectories before any score is observed, and evaluative taste is the gap between the proxy score and the quality it should match. We instantiate the vocabulary on recent autoresearch systems to illustrate that it covers designs that differ widely in structure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。