大模型评估会受之前对话情绪影响，越不确定越易被带偏。
AMEL: Accumulated Message Effects on LLM Judgments
- 测试相同内容在正/负评价后，模型判断倾向变化
- 负面历史导致偏差是正面的1.52倍，且对模糊问题影响更大
- 换新对话框可避免偏差，适合评估系统设计者参考
大型语言模型常被用作自动评估工具：审查代码、内容审核或评分输出，通常多个项目通过同一对话流。我们探究先前对话历史的极性是否会影响后续判断，这种现象称为大模型判断中的累积信息效应（AMEL）。通过对5家供应商的12个模型（含OpenAI、Anthropic、Google、DeepSeek及4个开源模型）共84,088次API调用进行实验，将相同测试项置于孤立状态或此前充满正向/负向评价的历史之后。结果显示，模型判断会向对话主流极性倾斜（d = -0.17，p < 10^-53）。该效应集中在模型本身基线不确定的项目上（高熵项目d = -0.36，确定性项目d = -0.15）。偏差不随上下文长度增加而增长（5轮与50轮效果相同，Spearman |r| < 0.01；OLS斜率p = 0.80）。存在负向不对称性：负向历史引起的偏差是正向的1.52倍（t = 13.03，p < 10^-36，n = 2,733）。模型规模虽缓解但无法根除该问题（Anthropic：Haiku -0.22 → Opus -0.17；OpenAI：Nano -0.34 → GPT-5.2 -0.17）。三次补充实验表明：概率分布连续偏移而非突变；负向不对称兼具词级与语义成分；位置无关性：5个偏差回合在50轮历史中任意位置均产生同等影响。最简修复方案是每项独立新上下文；批量处理时需平衡历史极性。
原文摘要 · Abstract (English)
Large language models are routinely used as automated evaluators: to review code, moderate content, or score outputs, often with many items passing through one conversation. We ask whether the polarity of prior conversation history biases subsequent judgments, an effect we call the accumulated message effect on LLM judgments (AMEL). Across 84,088 API calls to 12 models from 5 providers (OpenAI, Anthropic, Google, DeepSeek, and four open-source models), we present identical test items in isolation or following histories saturated with predominantly positive or negative evaluations. Models shift toward the conversation's prevailing polarity (d = -0.17, p < 10^-53). The effect concentrates on items where the model is genuinely uncertain at baseline (d = -0.36 for high-entropy items, vs d = -0.15 when the baseline is deterministic). Bias does not grow with context length: 5 prior turns and 50 produce the same shift (Spearman |r| < 0.01; OLS slope p = 0.80). And there is a negativity asymmetry: paired per item, negative histories induce 1.52x more bias than positive (t = 13.03, p < 10^-36, n = 2,733). Scaling helps but does not solve it (Anthropic: Haiku -0.22 to Opus -0.17; OpenAI: Nano -0.34 to GPT-5.2 -0.17). Three follow-ups narrow the mechanism. The token probability distribution shifts continuously, not at a threshold. The negativity asymmetry has both token-level and semantic components, though attributing the balance is exploratory at our sample sizes. Position does not matter: five biased turns anywhere in a 50-turn history produce the same shift. The simplest fix for evaluation pipelines is a fresh context per item; when batching is unavoidable, balancing the history helps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。