测试大模型对意大利政党的政治倾向,发现其评价受提示方式影响明显。
Who Would You Vote For? Auditing Political Alignment in LLMs: An Italian Case-Study

- 设计多维度评估框架,让模型评价政党与领导人
- 不同模型评价不一致,且对提示敏感
- 适合关注AI政治偏见的研究者和政策制定者
随着用户在选举期间越来越多地依赖大语言模型获取政治信息和建议,这些系统表达的政治偏好已成为公众关注焦点。已有研究显示,与大模型的互动可能影响用户的政治态度与选择,引发对其自身如何评估政治人物的疑问。本文探讨大模型是否以及如何表现出对政党与领导人的偏好。我们提出一种系统化、可复现的审计框架,通过九项标准对多个大模型进行提示,评估其对意大利政党与领导人的表现。不试图推断模型的‘真实’政治立场,而是关注其可观察行为:评价的一致性、模型间差异、拒绝回答率及提示形式的影响。进一步考察当模型被要求扮演不同角色时,评价如何变化。通过意大利案例研究,系统分析了大模型生成的政治评价结果。
原文摘要 · Abstract (English)
As users increasingly turn to Large Language Models (LLMs) for information and advice on political matters, particularly during election periods, the political preferences expressed by these systems have become a matter of public interest. Prior research has shown that interactions with LLMs can influence users' political attitudes and choices, raising questions about how these models themselves evaluate political actors. In this paper, we investigate whether and how LLMs express preferences toward political parties and political leaders. We introduce a systematic and reproducible auditing framework in which multiple LLMs are prompted to evaluate parties and leaders across nine criteria. Rather than attempting to infer the models' "true" political beliefs, we focus on their observable behavior, examining consistency across evaluations, differences between models, refusal rates, and sensitivity to prompt formulation. We further investigate how these evaluations vary when models are instructed to adopt different personas. We demonstrate the framework through an Italian case study, providing a systematic analysis of LLM-generated political evaluations on italian parties and leaders.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。