让无法比较的模型目标自动可比，轻松选出最优模型
COPA: Comparing the incomparable in multi-objective model evaluation
- 用累积函数和相对排名统一不同指标尺度
- 支持用户偏好定制，自动聚合多目标并导航帕累托前沿
- 适用于公平学习、自动化机器学习等场景
在机器学习中，常需从数百个训练好的模型中根据准确率、鲁棒性、公平性或可扩展性等多个目标进行选择。然而，这些目标常以不同单位和量纲衡量，难以直接比较、聚合与权衡，导致决策耗时且依赖专家经验。本文提出COPA方法，通过相对排名近似累积函数，将不可比的目标转化为可比形式，实现自动归一化与聚合，并匹配用户特定偏好，帮助使用者系统性地导航帕累托前沿。我们在公平学习、领域泛化、AutoML及基础模型等多个领域验证了COPA在模型选择与基准测试中的有效性，证明其优于传统归一化与聚合方式。
原文摘要 · Abstract (English)
In machine learning (ML), we often need to choose one among hundreds of trained ML models at hand, based on various objectives such as accuracy, robustness, fairness or scalability. However, it is often unclear how to compare, aggregate and, ultimately, trade-off these objectives, making it a time-consuming task that requires expert knowledge, as objectives may be measured in different units and scales. In this work, we investigate how objectives can be automatically normalized and aggregated to systematically help the user navigate their Pareto front. To this end, we make incomparable objectives comparable using their cumulative functions, approximated by their relative rankings. As a result, our proposed approach, COPA, can aggregate them while matching user-specific preferences, allowing practitioners to meaningfully navigate and search for models in the Pareto front. We demonstrate the potential impact of COPA in both model selection and benchmarking tasks across diverse ML areas such as fair ML, domain generalization, AutoML and foundation models, where classical ways to normalize and aggregate objectives fall short.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。