兼顾准确率与校准度，高效寻找最优示例组合。
COM-BOM: Bayesian Exemplar Search for Efficiently Exploring the Accuracy-Calibration Pareto Frontier
- 用贝叶斯优化同时优化预测准确率和模型校准度。
- 在多任务上实现准确率与校准度的协同提升，仅需少量API调用。
- 适合关注模型可信性与部署安全性的研究人员。
在上下文学习中，选择最优示例集对性能至关重要。然而，现有示例搜索方法仅关注预测准确率，严重忽视了模型校准——这一影响可信性与安全部署的关键因素。本文将示例选择建模为多目标优化问题，明确同时最大化预测准确率与最小化期望校准误差。我们提出一种样本高效的组合型贝叶斯优化算法（COM-BOM），用于高效探索准确率-校准度帕累托前沿。在未饱和的MMLU-Pro基准多个任务上的评估表明，COM-BOM在联合优化两个目标上优于或匹配基线方法，且所需大语言模型API调用次数最少。
原文摘要 · Abstract (English)
Selecting an optimal set of exemplars is critical for good performance of in-context learning. However, prior exemplar search methods narrowly optimize for predictive accuracy, critically neglecting model calibration--a key determinant of trustworthiness and safe deployment. In this paper, we formulate exemplar selection as a multi-objective optimization problem, explicitly targeting both the maximization of predictive accuracy and the minimization of expected calibration error. We solve this problem with a sample-efficient Combinatorial Bayesian Optimization algorithm (COM-BOM) to find the Pareto front that optimally trades off the two objectives of accuracy and calibration. We evaluate COM-BOM on multiple tasks from unsaturated MMLU-Pro benchmark and find that COM-BOM beats or matches the baselines at jointly optimizing the two objectives, while requiring a minimal number of LLM API calls.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。