让预测评估更贴近实际应用效果,自动学习最优评价标准。
Aligning the Evaluation of Probabilistic Predictions with Downstream Value
- 用神经网络学习加权评分规则,使评估与下游任务表现对齐。
- 在回归任务中验证,能有效缩小预测质量与实际价值间的差距。
- 适合需要快速评估多个复杂任务的系统设计者使用。
每个预测最终都会用于下游任务,因此评估预测质量应结合其实际应用效果。现有方法或依赖多个特定任务指标(分析负担重),或需预先设定成本结构(假设不现实)。本文将此偏差视为评估对齐问题,提出一种数据驱动的方法,通过神经网络学习与下游表现一致的代理评估函数。基于恰当评分准则理论,探索保持准则性质的评分变换,采用由神经网络参数化的加权评分规则,权重自动学习以匹配下游任务性能。该方法支持快速、可扩展的多任务评估,尤其适用于权重复杂或未知的场景。通过合成数据和真实数据的回归实验验证,证明其在模块化预测系统中能有效弥合预测评估与实际效用之间的鸿沟。
原文摘要 · Abstract (English)
Every prediction is ultimately used in a downstream task. Consequently, evaluating prediction quality is more meaningful when considered in the context of its downstream use. Metrics based solely on predictive performance often diverge from measures of real-world downstream impact. Existing approaches incorporate the downstream view by relying on multiple task-specific metrics, which can be burdensome to analyze, or by formulating cost-sensitive evaluations that require an explicit cost structure, typically assumed to be known a priori. We frame this mismatch as an evaluation alignment problem and propose a data-driven method to learn a proxy evaluation function aligned with the downstream evaluation. Building on the theory of proper scoring rules, we explore transformations of scoring rules that ensure the preservation of propriety. Our approach leverages weighted scoring rules parametrized by a neural network, where weighting is learned to align with the performance in the downstream task. This enables fast and scalable evaluation cycles across tasks where the weighting is complex or unknown a priori. We showcase our framework through synthetic and real-data experiments for regression tasks, demonstrating its potential to bridge the gap between predictive evaluation and downstream utility in modular prediction systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。