研究如何设计提示词让大模型更真实地模拟不同社会群体。
The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Models
- 用访谈式提示和姓名引导提升模拟真实性。
- 小模型反而比大模型更少刻板印象。
- 适合做社会偏见研究或公平性评估的团队参考。
人格提示在大语言模型中被广泛用于模拟不同社会人口群体的观点。然而，提示词的构建方式会显著影响结果，引发对模拟真实性的担忧。我们使用五个开源大模型，系统评估了角色扮演格式与人口特征预设策略对15个交叉人口群体在开放与封闭式任务中模拟表现的影响。结果表明，大模型在模拟边缘化群体时表现不佳，而采用访谈式提示和基于姓名的预设可有效减少刻板印象并提升一致性。令人意外的是，较小的模型如OLMo-2-7B在多数任务上优于更大的Llama-3.3-70B。研究为设计更具真实性和公平性的社会人口提示提供了实用指导。
原文摘要 · Abstract (English)
Persona prompting is increasingly used in large language models (LLMs) to simulate views of various sociodemographic groups. However, how a persona prompt is formulated can significantly affect outcomes, raising concerns about the fidelity of such simulations. Using five open-source LLMs, we systematically examine how different persona prompt strategies, specifically role adoption formats and demographic priming strategies, influence LLM simulations across 15 intersectional demographic groups in both open- and closed-ended tasks. Our findings show that LLMs struggle to simulate marginalized groups but that the choice of demographic priming and role adoption strategy significantly impacts their portrayal. Specifically, we find that prompting in an interview-style format and name-based priming can help reduce stereotyping and improve alignment. Surprisingly, smaller models like OLMo-2-7B outperform larger ones such as Llama-3.3-70B. Our findings offer actionable guidance for designing sociodemographic persona prompts in LLM-based simulation studies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。