评测大模型代码生成中的偏见，提出量化方法并测试缓解策略。
Measuring and Mitigating Bias in Code Generated by Large Language Models
- 构建双指标框架，用CBS和ACR量化代码偏见与属性影响。
- 四种轻量级策略均无法完全消除偏见，尤其在不同数据集上仍显著存在。
- 适合关注AI代码安全与公平性的开发者、研究者参考。
大型语言模型（LLMs）在自然语言生成中广泛应用，正越来越多用于代码生成任务。然而，其生成结果中的偏见问题仍备受关注。本文聚焦主流代码生成工具GPT-4o和Gemini，提出一个评估偏见的框架，重点考察受保护属性、提示词设计及网页搜索能力的影响。采用代码偏见分数（CBS）和属性变化率（ACR）两个指标，分别量化偏见程度与属性影响强度。进一步研究了四种轻量级缓解策略：少样本提示（Few-Shot）、思维链（Chain-of-Thought）、少样本思维链（Few-Shot Chain-of-Thought）及多智能体（Multi-agent），旨在降低生成代码中的偏见。结果显示，即使应用这些策略，偏见在不同受保护属性和数据集上依然普遍存在，表明当前方法仍不足以有效解决人工智能驱动代码生成中的偏见问题，亟需更有效的干预机制。
原文摘要 · Abstract (English)
Large language models (LLMs) are widely recognised for their applications in natural language generation and are increasingly used for code generation tasks. However, concerns about bias in their generated outputs remain significant. This paper focuses on GPT-4o and Gemini, mainstream tools for code generation, and proposes a framework for evaluating bias in LLM-generated code, specifically examining the influence of protected attributes, prompts and web-search capability. We use two metrics: the code bias score (CBS) and the attribute change ratio (ACR), to quantify the prevalence of bias and the degree of influence of different attributes, respectively. In addition, we investigate four lightweight mitigation strategies: Few-Shot, Chain-of-Thought, Few-Shot Chain-of-Thought, and Multi-agent, aimed at mitigating bias in generated code. Our findings reveal that bias remains prevalent across different protected attributes and datasets even after applying mitigation strategies, highlighting the need for more effective approaches to reduce bias in AI-driven code generation systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。