让多个大模型的置信度在运行时自动校准，提升协作决策准确率。
MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination
- 在线学习各模型置信度校准因子，无需模型访问或重新训练
- 在44,000+观测中，显著缩小分布偏移下的置信度误差
- 适合需要可解释性、动态模型池的多智能体系统部署
基础模型池越来越多地作为协作系统中的黑箱响应者使用，协调器需决定信任哪个响应。原始自报告置信度是自然信号，但跨模型不可比，且在分布偏移下设计时校准会过时。本文研究多模型协作中的运行时置信度校准，通过部署结果在线学习每模型校准，无需模型访问、无保留校准数据、无需重训练。在18个开源大模型、8个基准和4.4万+观测上发现，在线适应是普遍特性：简单同信息在线校准器能有效弥补冻结设计校准的差距，遗忘策略是关键设计轴。提出MARGIN（多智能体运行时评分通过增量归一化），采用对称指数加权更新与收缩融合，维护每模型每置信带乘法因子。虽非在线方法中预期校准误差最优，但提供可解释的置信带信任因子，支持冷启动与返回模型行为，兼容动态池，并为固定策略非策略性代理提供对称更新保证。实证显示，原始口头置信度在困难代码生成任务中是弱或误导的成对选择信号，而在线校准显著提升成对分辨与多模型选择能力。还评估了延迟反馈与仅选答反馈；后者严重降低所有同信息在线方法，包括MARGIN。运行时校准充当异构基础模型池的协调层，MARGIN是实用且可检查的实现。
原文摘要 · Abstract (English)
Foundation-model pools are increasingly used as black-box responders in coordinated systems where a coordinator must decide which response to trust. Raw self-reported confidence is the natural signal, but is not comparable across models and becomes stale under distribution shift when corrected only at design time. We study runtime confidence calibration for multi-model coordination, where per-model corrections are learned online from deployment outcomes with no model access, no held-out calibration data, and no retraining. Across 18 open-weight foundation models, 8 benchmarks, and over 44,000 observations, we find that online adaptation is a family property: simple same-information online calibrators close most of the calibration gap left by frozen design-time methods under shift, and the forgetting schedule is the dominant design axis. We present MARGIN (Multi-Agent Runtime Grading via Incremental Normalisation), a structured member of this family that maintains per-model, per-confidence-band multiplicative factors using symmetric exponentially weighted updates and shrinkage blending. MARGIN does not dominate the online family on expected calibration error (ECE) under abrupt shift. Its value lies in interpretable confidence-band trust factors, defined cold-start and returning-model behaviour, dynamic-pool support, and a scoped symmetric-update guarantee for fixed-policy non-strategic agents. Empirically, raw verbalized confidence is a weak or misleading pairwise selection signal on hard code-generation tasks, while online calibration substantially improves pairwise resolution and multi-model selection. We also evaluate delayed and selected-answer-only feedback; the latter materially degrades every same-information online method, MARGIN included. Runtime calibration acts as a coordination layer for heterogeneous foundation-model pools, and MARGIN is a practical inspectable instantiation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。