用大模型生成真实多智能体知识工作数据集，解决隐私与多样性难题。
Using Large Language Models to Generate Authentic Multi-agent Knowledge Work Datasets
- 构建多智能体系统，由大模型协作生成文档与行为轨迹。
- 53%生成文档与74%真实文档被人类评为真实可信。
- 数据可共享无隐私风险，适合评估知识助手系统。
当前公开的知识工作数据集缺乏多样性、详尽标注及用户与文档的上下文信息，阻碍了知识工作辅助系统的客观评估与优化。由于真实场景数据收集成本高且需隐私保护，此类数据难以获取。为此，我们提出一种可配置的多智能体知识工作数据集生成器，模拟智能体间协作生成大模型文档及配套数据痕迹，并将配置信息或模拟过程中生成的背景信息统一存入知识图谱。生成的数据集可安全共享，无隐私或保密顾虑。实验显示，人类评估者认为53%的生成文档和74%的真实文档具有真实性，验证了方法的有效性。同时，我们分析了评估者反馈中的真实性标准，提出了针对常见问题的改进方向。
原文摘要 · Abstract (English)
Current publicly available knowledge work data collections lack diversity, extensive annotations, and contextual information about the users and their documents. These issues hinder objective and comparable data-driven evaluations and optimizations of knowledge work assistance systems. Due to the considerable resources needed to collect such data in real-life settings and the necessity of data censorship, collecting such a dataset appears nearly impossible. For this reason, we propose a configurable, multi-agent knowledge work dataset generator. This system simulates collaborative knowledge work among agents producing Large Language Model-generated documents and accompanying data traces. Additionally, the generator captures all background information, given in its configuration or created during the simulation process, in a knowledge graph. Finally, the resulting dataset can be utilized and shared without privacy or confidentiality concerns. This paper introduces our approach's design and vision and focuses on generating authentic knowledge work documents using Large Language Models. Our study involving human raters who assessed 53% of the generated and 74% of the real documents as realistic demonstrates the potential of our approach. Furthermore, we analyze the authenticity criteria mentioned in the participants' comments and elaborate on potential improvements for identified common issues.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。