arXiv:2409.09045cs.CYcs.AI2024-09被引 10

用大模型预测欧洲议会选举,发现结果受国家语言背景影响,准确性不均。

United in Diversity? Contextual Biases in LLM-Based Predictions of the 2024 European Parliament Elections

  • 用3个大模型根据2.6万选民信息预测投票行为
  • 预测准确率在不同国家和语言间差异显著
  • 需详细态度信息才有效,不适合普遍应用

基于大语言模型(LLMs)的合成样本被视作替代人类调查的高效方式,前提是其训练数据包含人类态度与行为信息。然而,由于训练与微调数据可能无法代表多样化语境,这些合成样本可能存在偏差,进而加剧研究、政策与社会中的既有偏见。本研究通过预测2024年欧洲议会选举结果,检验大模型在个体层面公众意见预测中是否存在情境依赖性偏差。我们向三个大模型提供26,000名合格欧洲选民的个体背景信息,要求其预测每个人的投票行为,并与实际结果对比。结果显示,大模型对未来的投票行为预测总体表现不佳,准确率在不同国家和语言情境下分布不均,且需要提示中包含详尽的态度信息才能提升效果。研究强调了大模型合成样本在公众意见预测中的局限性,并有助于理解与缓解大模型发展及其在计算社会科学中应用中的不平等。

原文摘要 · Abstract (English)

"Synthetic samples" based on large language models (LLMs) have been argued to serve as efficient alternatives to surveys of humans, assuming that their training data includes information on human attitudes and behavior. However, LLM-synthetic samples might exhibit bias, for example due to training data and fine-tuning processes being unrepresentative of diverse contexts. Such biases risk reinforcing existing biases in research, policymaking, and society. Therefore, researchers need to investigate if and under which conditions LLM-generated synthetic samples can be used for public opinion prediction. In this study, we examine to what extent LLM-based predictions of individual public opinion exhibit context-dependent biases by predicting the results of the 2024 European Parliament elections. Prompting three LLMs with individual-level background information of 26,000 eligible European voters, we ask the LLMs to predict each person's voting behavior. By comparing them to the actual results, we show that LLM-based predictions of future voting behavior largely fail, their accuracy is unequally distributed across national and linguistic contexts, and they require detailed attitudinal information in the prompt. The findings emphasize the limited applicability of LLM-synthetic samples to public opinion prediction. In investigating their contextual biases, this study contributes to the understanding and mitigation of inequalities in the development of LLMs and their applications in computational social science.

大模型选举预测偏见分析公众意见

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。