LLM判断结果受答题格式影响，二选一更易出负面结论。
Systematic Bias in Large Language Models: Discrepant Response Patterns in Binary vs. Continuous Judgment Tasks
- 对比二元与连续评分格式，发现模型倾向在二元格式中给出负面判断。
- 在价值观判断和情感分析任务中，二元格式的负面判断率显著更高。
- 研究提醒：设计决策类任务时需注意格式对结果的系统性干扰。
大型语言模型（LLMs）在心理文本分析和自动化决策流程中应用日益广泛，但其可靠性因训练过程带来的潜在偏见而受质疑。本研究考察了不同回答格式（二元与连续）对LLM判断的系统性影响。在价值陈述判断任务和文本情感分析任务中，我们让多个模型（包括开源与商业模型）模拟人类反应，测试两种格式的表现。结果发现，所有模型均呈现一致的负向偏差：在二元格式下，模型更倾向于输出“负面”判断，相较连续格式。控制实验进一步验证该现象在两类任务中均成立。研究强调，在应用LLM进行决策任务时，必须考虑响应格式的影响，因为任务设计的微小变化可能引入系统性偏差。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly used in tasks such as psychological text analysis and decision-making in automated workflows. However, their reliability remains a concern due to potential biases inherited from their training process. In this study, we examine how different response format: binary versus continuous, may systematically influence LLMs' judgments. In a value statement judgments task and a text sentiment analysis task, we prompted LLMs to simulate human responses and tested both formats across several models, including both open-source and commercial models. Our findings revealed a consistent negative bias: LLMs were more likely to deliver "negative" judgments in binary formats compared to continuous ones. Control experiments further revealed that this pattern holds across both tasks. Our results highlight the importance of considering response format when applying LLMs to decision tasks, as small changes in task design can introduce systematic biases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。