不同社会人口线索影响大模型个性化表现，单一线索不可靠。
One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalization
- 用六种社会人口线索测试大模型响应差异
- 同一用户不同线索导致回答差异显著，可能改变偏见结论
- 提醒避免依赖单一显性线索做公平性判断
通过社会人口子群对大模型进行个性化可提升用户体验，但也可能引入或加剧群体间的偏见与不公平。以往研究常使用单一线索（如用户名或明确属性）来提示模型人格，但忽略了提示变化对模型的影响，以及真实交互中某些线索的稀有性（外部有效性不足）。本文在七款开源及专有大模型上，针对四个写作与建议任务，对比了六种常用人格线索的表现。尽管各线索总体相关性高，但生成结果存在显著差异，可能改变关于人格诱导差异和偏见的结论。因此，我们警告不应仅基于单一线索做出判断，尤其当线索过于明显且缺乏外部有效性时。
原文摘要 · Abstract (English)
Personalization of LLMs by sociodemographic subgroup often improves user experience, but can also introduce or amplify biases and unfair outcomes across groups. Prior work has employed so-called personas, sociodemographic user attributes conveyed to a model, to study bias in LLMs by relying on a single cue to prompt a persona, such as user names or explicit attribute mentions. This disregards LLM sensitivity to prompt variation and the rarity of some cues in real interactions (external validity). We compare six commonly used persona cues across seven open and proprietary LLMs on four writing and advice tasks. While cues are overall highly correlated, they produce substantial variance in responses across personas that can change findings on persona-induced differences and bias. We therefore caution against claims based on single persona cues, especially when they are overly explicit and have low external validity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。