用好提示词，大模型能更准捕捉人类判断
LLMs Can Better Capture Human Judgments--With the Right Prompts
- 通过提示模型报告标准差和比例，还原人类响应全貌
- 清晰的场景描述使模型与人类判断对齐度提升
- 适合研究伦理判断、人机对齐的学者参考
大语言模型是否难以捕捉人类判断？常见观点认为其无法反映响应分布全貌，且对表述变化敏感。本文通过简单提示策略缓解这些问题。在包含144个美国代表性道德情景及来自32国的国际社会调查项目中38项性别角色信念数据集上，我们发现：要求模型输出标准差和响应比例，可更好还原人类完整反应分布；确保情景对人类清晰（以人类困惑评分反映）能显著提升模型对齐度，且模型可有效追踪人类困惑水平。同时，模型对自己的错误估计校准不佳，但能较好预测人类反应变异性。结果表明，优化提问方式可显著提升模型回答质量。
原文摘要 · Abstract (English)
Are large language models (LLMs) bad at capturing human judgment? Two commonly stated limitations are that LLMs fail to capture full distributions of responses, and that their judgments are unstable across wording variations. We demonstrate simple prompting strategies that mitigate these limitations. Across two datasets--a U.S.-representative set of 144 moral scenarios and 38 moral beliefs from the International Social Survey Programme's Family and Changing Gender Roles module covering 32 countries--we show how simple elicitation techniques help improve AI-human alignment. First, prompting models to report standard deviations and response proportions recovers the full range of human responses better than common strategies. Second, ensuring scenarios are clear to human participants--as reflected in human confusion ratings--boosts model alignment, and LLMs can track human confusion ratings. At the same time, we find that LLMs' estimates of their own error are poorly calibrated, though they can predict human variability relatively well. These results suggest that asking better questions to LLMs can yield better answers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。