让大模型回顾自己之前的回答，能有效降低偏见。
B-score: Detecting biases in large language models using response history
- 通过多轮对话让模型自检回答，减少偏见。
- 在随机类问题上，模型自纠偏见效果显著。
- 提出B-score指标，可精准识别多种类型偏见。
大型语言模型常表现出强烈偏见，例如对女性的偏见或对数字7的偏好。我们研究了在多轮对话中，允许模型观察自身先前回答时，是否能输出更少偏见的答案。针对不同类型的提问，我们设计了涵盖9个主题的测试集，分为三类：主观性、随机性与客观性问题。有趣的是，模型在面对需要随机、无偏答案的问题时，能通过多轮对话自我纠正偏见。为此，我们提出B-score这一新指标，能有效检测主观、随机、简单和复杂问题中的偏见。在MMLU、HLE和CSQA数据集上，使用B-score显著提升了模型答案验证的准确率，优于仅依赖口头置信度或单轮回答频率的方法。代码与数据已公开于https://b-score.github.io。
原文摘要 · Abstract (English)
Large language models (LLMs) often exhibit strong biases, e.g, against women or in favor of the number 7. We investigate whether LLMs would be able to output less biased answers when allowed to observe their prior answers to the same question in a multi-turn conversation. To understand which types of questions invite more biased answers, we test LLMs on our proposed set of questions that span 9 topics and belong to three types: (1) Subjective; (2) Random; and (3) Objective. Interestingly, LLMs are able to "de-bias" themselves in a multi-turn conversation in response to questions that seek an Random, unbiased answer. Furthermore, we propose B-score, a novel metric that is effective in detecting biases to Subjective, Random, Easy, and Hard questions. On MMLU, HLE, and CSQA, leveraging B-score substantially improves the verification accuracy of LLM answers (i.e, accepting LLM correct answers and rejecting incorrect ones) compared to using verbalized confidence scores or the frequency of single-turn answers alone. Code and data are available at: https://b-score.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。