提出可处理任意优先关系图的新型模型评分方法。
Generalized Priority-Aware Shapley Value
- 基于加权有向图定义新评分方法，允许优先关系循环
- 在聊天机器人偏好图上验证，不同优先设置导致明显不同评分结果
- 适合需处理复杂人类偏好或多重标准评估的场景
Shapley值及其优先感知扩展广泛用于机器学习中的估值，但现有方法要求成对优先关系为二元且无环，这一限制在真实数据（如聚合的人类偏好和多准则比较）中常被违反。本文提出广义优先感知Shapley值（GPASV），一种定义在任意加权有向优先图上的随机排序值，其中成对边仅惩罚而非禁止顺序违规。GPASV涵盖多种经典模型作为边界情况。我们通过公理化框架建立其理论基础，开发了相应的计算方法，并引入一种优先扫掠诊断工具，扩展了已有PASV方法。将GPASV应用于包含循环结构的Chatbot Arena偏好图上，揭示优先感知估值并非简单一键操作：成对图优先关系与个体软优先之间的不同平衡，会产生显著不同的模型评分结果。
原文摘要 · Abstract (English)
Shapley value and its priority-aware extensions are widely used for valuation in machine learning, but existing methods require pairwise priority to be binary and acyclic, a restriction spectacularly violated in real-data examples such as aggregated human preferences and multi-criterion comparisons. We introduce the generalized priority-aware Shapley value (GPASV), a random order value defined on arbitrary directed weighted priority graphs, in which pairwise edges penalize rather than forbid order violations. GPASV covers a range of classical models as boundary cases. We establish GPASV through an axiomatic characterization, develop the associated computational methods, and introduce a priority sweeping diagnostic extending PASV's. We apply GPASV to LLM ensemble valuation on the cyclic Chatbot Arena preference graph, illustrating that priority-aware valuation is not a one-button operation: different balances of pairwise graph priority versus individual soft priority produce substantively different valuations of the same data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。