让大模型代理协作搜集信息，模拟真实多人协作场景。
LM Agents for Coordinating Multi-User Information Gathering
- 用大模型代理模拟多用户分工协作，主动找人问问题。
- 在表格问答和文档生成任务中，最高准确率达78.3%。
- 适合研究多智能体协作与真实工作流的学者使用。
本文提出PeopleJoin，一个用于评估大语言模型驱动的协作式问题解决能力的基准。面对用户请求时，PeopleJoin代理需识别可能提供帮助的队友，与其对话获取信息，并最终为原始用户提供有用答案或摘要。该基准包含两个评估领域：PeopleJoin-QA（针对表格数据的问题）和PeopleJoin-DocCreation（针对文档创作任务），二者均源自现有NLP基准，但信息分布于2至20名合成用户组成的虚拟组织中，模拟真实多用户协作环境。我们实现了多种主流大模型代理架构，在完成任务的准确性与效率上进行评估，并揭示了可基于此基准开展的新研究问题。
原文摘要 · Abstract (English)
This paper introduces PeopleJoin, a benchmark for evaluating LM-mediated collaborative problem solving. Given a user request, PeopleJoin agents must identify teammates who might be able to assist, converse with these teammates to gather information, and finally compile a useful answer or summary for the original user. PeopleJoin comprises two evaluation domains: PeopleJoin-QA, focused on questions about tabular data, and PeopleJoin-DocCreation, focused on document creation tasks. The two domains are adapted from existing NLP benchmarks for database question answering and multi-document summarization; here, however, the information needed to complete these tasks is distributed across synthetic ``organizations'' of 2--20 users, simulating natural multi-user collaboration scenarios. We implemented several popular LM agent architectures, evaluating their accuracy and efficiency at completing tasks, and highlight new research questions that can be studied using PeopleJoin.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。