通过显式注入最佳实践,显著降低AI在软件工程决策中的提示偏见。
Mitigating Prompt-Induced Cognitive Biases in General-Purpose AI for Software Engineering

- 显式提取并注入软件工程最佳实践作为推理前提,对抗隐性偏见。
- 新方法使整体偏见敏感度平均降低51%(p<0.001)。
- 识别出高偏见风险的语言模式,指导实际应用边界。
提示诱导的认知偏见是指通用人工智能(GPAI)系统因输入措辞偏差(如框架、锚定)而产生的决策变化,而非任务逻辑本身所致。在以自然语言表达需求的软件工程(SE)决策支持中,微小的措辞差异(如流行度暗示或结果透露)可能引导GPAI走向次优决策。我们通过PROBE-SWE——一个动态基准测试,对比同一SE困境的有偏与无偏版本,在控制逻辑和难度的前提下,研究八类典型偏见(锚定、可得性、从众、确认、框架、后见之明、超前折扣、过度自信)。测试常见提示工程技术(如思维链、自我去偏)在低成本GPAI系统上的效果,发现各偏见层面均无统计显著缓解。由此提出:偏见特征会截断背景公理与推理假设的显式化过程。为此,我们设计一种端到端方法,在回答前主动提取最佳实践并注入命题推理线索,使整体偏见敏感度平均下降51%(p < .001)。最后,通过主题分析揭示与偏见敏感度相关的语言模式,明确GPAI在特定情境下不适用,并指明未来应对方向。
原文摘要 · Abstract (English)
Prompt-induced cognitive biases are changes in a general-purpose AI (GPAI) system's decisions caused solely by biased wording in the input (e.g., framing, anchors), not task logic. In software engineering (SE) decision support (where problem statements and requirements are natural language) small phrasing shifts (e.g., popularity hints or outcome reveals) can push GPAI models toward suboptimal decisions. We study this with PROBE-SWE, a dynamic benchmark for SE that pairs biased and unbiased versions of the same SE dilemmas, controls for logic and difficulty, and targets eight SE-relevant biases (anchoring, availability, bandwagon, confirmation, framing, hindsight, hyperbolic discounting, overconfidence). We ask whether prompt engineering mitigates bias sensitivity in practice, focusing on actionable techniques that practitioners can apply off-the-shelf in real environments. Testing common strategies (e.g., chain-of-thought, self-debiasing) on cost-effective GPAI systems, we find no statistically significant reductions in bias sensitivity on a per-bias basis. We then adopt a Prolog-style view of the reasoning process: solving SE dilemmas requires making explicit any background axioms and inference assumptions (i.e., SE best practices) that are usually implicit in the prompt. So, we hypothesize that bias-inducing features short-circuit assumption elicitation, pushing GPAI models toward biased shortcuts. Building on this, we introduce an end-to-end method that elicits best practices and injects axiomatic reasoning cues into the prompt before answering, reducing overall bias sensitivity by 51% on average (p < .001). Finally, we report a thematic analysis that surfaces linguistic patterns associated with heightened bias sensitivity, clarifying when GPAI use is less advisable for SE decision support and where to focus future countermeasures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。