研究发现,选对人物属性能显著提升大模型预测调查回答的准确性。
When Persona Attributes Improve Population Alignment in Large Language Models
- 基于人类回答差异性判断何时使用人物属性提示更有效
- 在4个社会调查、6个模型、20项任务中验证了方法有效性
- 为大模型做问卷预测时选择属性提供实用指导
大语言模型(LLMs)正被用于预测调查样本中人类参与者的行为响应。近年来,人物属性提示(persona prompting)作为一种引导预训练大模型生成的方法逐渐兴起。该方法通过在提示中加入简短的人物描述(如社会人口学特征、态度或行为),使模型生成的回答更贴近真实人类反应。然而,现有研究结果不一致,部分甚至相互矛盾,仅少数发现属性选择至关重要,且增加属性数量未必提升效果。本文提出,调查问题的人类回答差异性可能是导致表现波动的原因之一。我们比较了不同属性选取方法在四份跨两国通用社会调查上的表现,涵盖六种主流大模型及每项调查中的二十个预测任务。研究揭示了人物提示在何种情况下更有效,并为基于人物提示的调查预测提供了新的方法评估依据。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly used to predict the responses of human participants in survey panels. Towards that goal, persona prompting has recently emerged as a technique to inform and align large pretrained language models. Persona prompting refers to the practice of using short textual descriptions of 'personas' in prompts to steer the LLM's generations. Personas describe individuals through different attributes such as their socio-demographics, attitudes, or behaviors, with the aim of aligning LLMs to produce responses that correlate with the corresponding human responses. Yet, recent work has produced mixed and partly conflicting results of persona prompting without clear patterns of success and failure. Among the few consistent findings is that the selection of persona attributes matters, and that using more attributes does not necessarily lead to better performance. It remains unclear how different attribute selection methods perform and how to choose among them. In this paper, we propose that observed human response variation of a survey question is a potential explanation for the mixed performance observed so far. In addition, we compare the performance of persona prompting associated with different methods for selecting persona attributes. We evaluate these methods on four different (general) social surveys across two countries, six LLMs, and twenty prediction tasks per survey. Our work helps to identify when persona prompting can be expected to be useful in survey prediction tasks, and provides new insights on the effectiveness of different attribute selection methods for LLM-based survey prediction using persona prompting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。