通过人类对中间步骤的偏好标注，提升生成式代理的社会行为真实性。
Step-Level Preference Learning for Generative Agents in Social Simulations
- 构建交互界面收集57000条人类对代理决策过程的逐步偏好数据。
- 在开放权重模型上进行偏好学习，显著提升行为协调性与社会适应性。
- 适合研究社交模拟、人机协作或强化学习可解释性的研究人员。
基于大语言模型的生成式代理通过长时程决策过程模拟人类行为，包括规划、记忆检索、反思和动作选择等中间步骤。然而，这些中间步骤的细粒度人类标注稀缺，现有代理也缺乏对中间决策的人类偏好支撑。为此，我们提出 extit{method}，一种交互式仿真界面，用于收集人类对代理决策轨迹的逐步偏好监督，形成包含57,000条细粒度标注的数据集。我们在该数据上对开放权重语言模型进行有监督微调与直接偏好优化，持续提升了模拟保真度、协作能力与交互质量，促使代理表现出更符合社会规范的行为。结果表明，逐步人类监督是改善局部决策质量与长时程代理行为的有效训练信号。
原文摘要 · Abstract (English)
Large language model (LLM)-based generative agents simulate human behavior through long-horizon decision-making processes that comprise intermediate steps such as planning, memory retrieval, reflection, and action selection. However, fine-grained human annotations of these intermediate steps remain scarce, and existing agents are not grounded in human preferences over such intermediate decisions. To address this gap, we introduce \method, an interactive simulation interface that enables us to collect step-level human preference supervision over agent decision trajectories, leading to a dataset of 57K fine-grained annotations. We conduct step-level preference learning on open-weight language models using supervised finetuning and direct preference optimization on this data, consistently improving simulation fidelity, coordination, and interaction quality, and inducing more socially effective agent behavior. Our results show that step-level human supervision is an effective training signal for improving both local decision quality and long-horizon agent behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。