arXiv:2605.00382cs.SEcs.AI2026-05被引 2

发现大模型代码生成存在严重社会偏见,提出新方法有效降低偏见并提升正确率。

Social Bias in LLM-Generated Code: Benchmark and Mitigation

论文配图:Social Bias in LLM-Generated Code: Benchmark and Mitigation
图 1 · 摘自论文原文
  • 构建7维度343个真实任务的基准,发现代码偏见最高达60.58%
  • 传统提示优化反而加剧偏见,结构化协作流程需明确角色职责才有效
  • 引入可插拔的公平性监控代理,不改流程即可降偏见65.1%且提正确率

大型语言模型(LLMs)在面向人类的应用中生成代码日益普遍,而此类应用对人口统计公平性至关重要。然而现有评估几乎仅关注功能正确性,忽视了代码中的社会偏见。基于我们之前的工作Solar,本文通过SocialBias-Bench——一个涵盖七个社会维度的343个真实编码任务的基准,对四个主流LLM进行了全面实证研究。结果显示所有模型均存在严重偏见,代码偏见分数高达60.58%。我们还发现,标准的提示级干预(如思维链推理、公平角色设定)反而会放大偏见。进一步研究显示,结构化的多智能体软件流程能减轻偏见,前提是早期角色能正确界定代码应考虑或排除的属性。但若在所有角色中加入显式公平指令,效果反而更差,表明责任分散问题未被解决。为此,我们提出公平性监控代理(FMA),一种可无缝集成至任意代码生成流水线的模块化组件。FMA分析任务描述以确定应考虑或限制的属性,通过迭代审查检测并纠正偏差,无需可执行测试套件。在全部343个任务上评估,相比仅使用开发代理,FMA将偏见降低65.1%,同时将功能正确率从75.80%提升至83.97%,优于所有其他对比方法。

原文摘要 · Abstract (English)

Large Language Models (LLMs) are increasingly deployed to generate code for human-centered applications where demographic fairness is critical. However, existing evaluations focus almost exclusively on functional correctness, leaving social bias in LLM-generated code largely unexamined. Extending our prior work on Solar, we conduct a comprehensive empirical study using SocialBias-Bench, a benchmark of 343 real-world coding tasks spanning seven demographic dimensions. We evaluate four prominent LLMs and find severe bias across all models, with Code Bias Scores reaching up to 60.58%. We further show that standard prompt-level interventions, such as Chain-of-Thought reasoning and fairness persona assignment, inadvertently amplify bias rather than reduce it. We then investigate whether structured multi-agent software process frameworks can improve fairness, finding that structured pipelines reduce bias when early roles correctly scope what the code should and should not consider. However, adding explicit fairness instructions to all agent roles produces worse outcomes than providing none, suggesting that diffused responsibility goes unaddressed. To address these limitations, we propose the Fairness Monitor Agent (FMA), a modular component that plugs into any existing code generation pipeline without modifying it. FMA analyzes the task description to determine which attributes should be considered or restricted, then detects and corrects violations through an iterative review process, without requiring an executable test suite. Evaluated on all 343 tasks, FMA reduces bias by 65.1% compared to a developer agent alone and improves functional correctness from 75.80% to 83.97%, outperforming all other studied approaches.

代码生成社会偏见公平性LLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。