构建249万真实用户提示集，测出大模型在政治议题上的偏见倾向。
IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance
- 基于3900个模板和212个真实议题生成海量提示
- 10个顶尖模型普遍存在议题偏见，倾向美国民主党观点
- 可扩展的评测基准，适合政策与伦理研究者使用
大型语言模型正帮助数百万用户撰写涉及多元议题的文本，但可能引发议题偏见问题——即模型仅呈现单一观点，影响用户认知。当前缺乏真实交互场景下的偏见测量工具。为此，我们构建IssueBench：一个包含249万条真实英文提示的基准，基于3900个模板（如“写一篇博客关于”）和212个真实政治议题（如“人工智能监管”）。利用该基准，我们发现10个领先大模型普遍存在议题偏见，且偏见模式高度相似；在部分议题上，所有模型均更接近美国民主党而非共和党选民立场。IssueBench可轻松拓展至其他议题、模板或任务。该工具为评估和应对大模型偏见提供了可复现、高保真的实证基础。
原文摘要 · Abstract (English)
Large language models (LLMs) are helping millions of users write texts about diverse issues, and in doing so expose users to different ideas and perspectives. This creates concerns about issue bias, where an LLM tends to present just one perspective on a given issue, which in turn may influence how users think about this issue. So far, it has not been possible to measure which issue biases LLMs manifest in real user interactions, making it difficult to address the risks from biased LLMs. Therefore, we create IssueBench: a set of 2.49m realistic English-language prompts to measure issue bias in LLM writing assistance, which we construct based on 3.9k templates (e.g. "write a blog about") and 212 political issues (e.g. "AI regulation") from real user interactions. Using IssueBench, we show that issue biases are common and persistent in 10 state-of-the-art LLMs. We also show that biases are very similar across models, and that all models align more with US Democrat than Republican voter opinion on a subset of issues. IssueBench can easily be adapted to include other issues, templates, or tasks. By enabling robust and realistic measurement, we hope that IssueBench can bring a new quality of evidence to ongoing discussions about LLM biases and how to address them.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。