校准评价器可减少大模型反馈中的偏好偏差传播。
Calibrating the Evaluator: Does Probability Calibration Mitigate Preference Coupling in LLM Agent Feedback Loops?
- 用概率校准改进评价器的判断，降低错误偏好传递。
- 实验显示耦合系数下降20%-49%，分布差异减少45%-67%。
- 适合关注模型评价可靠性与反馈循环优化的研究者。
当大语言模型代理通过评价器反馈调整行为时，评价器的系统性偏差会渗透到代理的学习策略分布中，这一现象称为评价器偏好耦合（EPC）。已有研究识别了该问题并建立了诊断框架，但未探索校准技术是否能缓解它。本文首次系统研究评价器校准作为缓解手段：对评价器的成对判断应用概率校准，以减少虚假偏好传播。在一项受控的被试内实验中（N=5），使用DeepSeek-V4-Pro作为执行器、GLM5.2作为评价器，对比标准二元TTRL（胜/负）与置信度校准的TTRL（概率加权更新），结果表明校准使耦合系数γ降低20%-49%，Jensen-Shannon散度减少45%-67%。对称学习率对照组确认该效果并非源于更新不对称性的降低。我们公开了校准后的TTRL协议，并建议其作为部署中轻量级缓解方案。
原文摘要 · Abstract (English)
When large language model (LLM) agents adapt their behavior through evaluator feedback, systematic evaluator biases propagate into the agent's learned strategy distribution - a phenomenon termed evaluator preference coupling. Prior work has documented this coupling and established a diagnostic framework (EPC) to measure it, but has not investigated whether calibration techniques can mitigate the effect. We present the first study of evaluator calibration as mitigation: applying probability calibration to the evaluator's pairwise judgments to reduce spurious preference propagation. In a controlled within-subjects experiment (N=5) comparing standard binary TTRL (win/loss) with confidence-calibrated TTRL (probability-weighted updates) using DeepSeek-V4-Pro as executor and GLM5.2 as evaluator, we find that calibration reduces the coupling coefficient gamma by 20-49% and Jensen-Shannon divergence by 45-67%. A symmetric-LR control confirms the effect is not due to reduced update asymmetry. We release the calibrated TTRL protocol and recommend it as a lightweight mitigation for LLM-as-judge deployment pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。