大模型对极右翼政党文本更困惑，暴露政治偏见。
Large Language Models are Perplexed by some Political Parties
- 用困惑度衡量模型对不同政党的公平性
- 极右翼政党文本困惑度高出社会民主派37%
- 偏见源于预训练，指令微调无法缓解
大型语言模型（LLMs）在政治领域应用日益广泛，但其政治公平性研究不足。我们通过困惑度评估公平性，认为公平模型应对所有政治群体给予同等概率。在覆盖37种语言的三个数据集上，测试十款模型发现，模型对极右翼与民族主义政党文本的困惑度显著高于社会民主党。该现象与先前翻译公平性研究一致，困惑度与下游翻译指标高度相关。该方法适用于基础模型及指令微调版本，两者表现高度相关，表明政治偏见主要源自预训练阶段，指令微调难以改善。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly used, including in political applications, but their political fairness has been little studied. We assess it using perplexity, posing that a fair model should give equal probability to all political groups. However, we find, across ten LLMs and three datasets covering 37 languages, that LLMs are more perplexed by the texts of far right and nationalist parties than of social-democratic parties. We find this to be consistent with previous work on translation fairness, to the point that perplexity correlates with downstream translation metrics. Our method is applicable to both base LLMs as well as their instruction-tuned counterpart, and we find that both are highly correlated, suggesting that the political fairness of LLMs stems from their pretraining, and is hardly affected by instruction-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。