提出标准化协议，量化大模型代理中评估者偏见的传播动态。
EPC: A Standardized Protocol for Measuring Evaluator Preference Dynamics in LLM Agent Systems
- 定义EPC四阶段隔离范式，规范实验配置与评估流程。
- 提供包含122次实验的参考快照，覆盖8种评估条件与多个模型版本。
- 支持第三方复现、跨时间比较，检测评估器更新导致的性能衰减。
当大语言模型代理在闭环中利用评估者反馈调整行为时，评估者偏见会通过代理策略分布传递——这一现象称为评估者偏好耦合。已有研究揭示了不同评估器家族和模型版本间的耦合现象，但缺乏标准化协议，难以实现第三方复现、跨评估器与时间点的结果对比，也无法检测专有评估器静默更新导致的测量衰减。本文提出EPC（评估者偏好耦合）协议，以RFC风格详细定义四阶段隔离范式，涵盖执行器与评估器配置、策略与任务设计、TTRL更新规则、指标计算（伽马、JSD、ECE、Brier）及输出格式。配套发布版本化参考快照v1.0：基于五个独立研究，涵盖8种评估条件（共122次唯一实验重复），涉及GPT-4o、Qwen、DeepSeek等模型，标注评估器版本、API端点与测量日期。快照具有时间限定性，所有值均依赖特定模型版本，预计随专有评估器更新而衰减。定义版本命名规范（vX.Y-Z），并提供使用指南，涵盖采用、解读与已知陷阱。协议、参考快照与实现代码均已开源。
原文摘要 · Abstract (English)
When LLM agents use evaluator feedback to adapt their behavior in closed loops, evaluator biases propagate through the agent's strategy distribution -- a phenomenon known as evaluator preference coupling. Prior work has documented coupling across multiple evaluator families and model versions, but the field lacks a standardized protocol that enables third-party researchers to (i) reproduce coupling measurements, (ii) compare results across evaluators and time points, and (iii) detect measurement decay as proprietary evaluators silently update. This paper provides the protocol. We specify EPC (Evaluator Preference Coupling) -- a detailed, RFC-style protocol specification for the four-phase isolation paradigm, covering executor and evaluator configuration, strategy and task design, the TTRL update rule, metric computation (gamma, JSD, ECE, Brier), and output schema. We accompany the protocol with a versioned Reference Snapshot v1.0: coupling measurements for eight evaluator conditions (N=122 unique experimental repetitions across GPT-4o, Qwen, DeepSeek, and others) derived from five independent studies, annotated with evaluator version identifiers, API endpoints, and measurement dates. The snapshot is explicitly time-bound: all values are conditional on specific model versions and are expected to decay as proprietary evaluators update. We define a versioning convention (vX.Y-Z, encoding protocol version, snapshot version, and evaluator generation) and provide a usage guide covering adoption, interpretation, and known pitfalls. The protocol, reference snapshot, and implementation code are released as open infrastructure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。