揭露大模型人格实验的代表性和透明度问题，提出改进方案
Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparency
- 分析63篇顶会论文，发现人格设定常缺关键用户信息
- 仅35%研究讨论人格代表性，多数只关注有限社会属性
- 提出透明度检查清单，提升实验生态效度与可复现性
合成人格实验已成为大语言模型对齐研究中的主流方法，但其人格代表性和生态效度在不同研究中差异显著。通过对2023至2025年间发表于顶级NLP与AI会议的63篇同行评审论文的综述，我们揭示了一个关键问题：任务目标与目标人群常被忽视，而个性化依赖于此。分析显示，用户表征存在显著差异，多数研究仅关注有限的社会人口学特征，且仅有35%讨论其大模型人格的代表性。基于此，我们提出一个人格透明度检查清单，强调代表性抽样、实证数据明确支撑以及增强生态效度。本工作既提供了当前实践的全面评估，也给出了提升人格化评估严谨性与生态效度的实用指南。
原文摘要 · Abstract (English)
Synthetic personae experiments have become a prominent method in Large Language Model alignment research, yet the representativeness and ecological validity of these personae vary considerably between studies. Through a review of 63 peer-reviewed studies published between 2023 and 2025 in leading NLP and AI venues, we reveal a critical gap: task and population of interest are often underspecified in persona-based experiments, despite personalization being fundamentally dependent on these criteria. Our analysis shows substantial differences in user representation, with most studies focusing on limited sociodemographic attributes and only 35% discussing the representativeness of their LLM personae. Based on our findings, we introduce a persona transparency checklist that emphasizes representative sampling, explicit grounding in empirical data, and enhanced ecological validity. Our work provides both a comprehensive assessment of current practices and practical guidelines to improve the rigor and ecological validity of persona-based evaluations in language model alignment research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。