arXiv:2605.09588cs.GTcs.AI2026-05

从少量反馈中高效选出最佳模型组合,节省评估成本。

Efficient Ensemble Selection from Binary and Pairwise Feedback

  • 用投票机制建模多模型组合选择,支持二元与成对反馈
  • 提出贪心算法,在保证性能前提下减少近一半查询次数
  • 适合需要低成本筛选大模型的工程团队使用

组织在多个任务域部署多个AI系统,但选择少数高性能模型组合需大量模型调用、基准测试和人工评估。本文将此问题建模为分布式多胜者投票:任务来自未知分布,每项任务产生对候选模型的反馈,委员会在任务上的价值由表现最好的成员决定。研究二元反馈(正确/错误)和成对反馈(偏好比较)。在二元情形下,目标为覆盖度,给出全量获取基线与匹配最坏情况查询下界,并设计失败条件贪心算法,在保持(1-1/e)性能保证的同时实现实例依赖的查询节省。在成对情形下,研究θ-优委员会,证明全信息优化存在PTAS但无EPTAS(在Gap-ETH假设下),且目标单调但非子模。由此提出加权序数覆盖松弛,具子模性并支持成对反馈下的失败条件贪心预言机;再通过有限族审计或极小极大封装将预言机转化为θ-型保证。还提供小规模LLM实验,验证预测查询节省及互补性在组合选择中的作用。

原文摘要 · Abstract (English)

Organizations increasingly deploy multiple AI systems across task domains, but selecting a small, high-performing ensemble can require costly model calls, benchmark runs, and human evaluation. We study this selection problem as a distributional variant of multiwinner voting: tasks are drawn from an unknown domain distribution, each task induces feedback over candidate experts, and a committee's value on a task is determined by its best-performing member. We analyze both binary feedback, for tasks with correct/incorrect outcomes, and pairwise feedback, for tasks where candidate outputs are compared by preference. In the binary setting, the induced objective is coverage. We give exhaustive-elicitation baselines and matching worst-case query lower bounds, and we design a failure-conditioned greedy algorithm that preserves the standard $(1-1/e)$ guarantee while obtaining instance-dependent query savings. In the pairwise setting, we study $θ$-winning committees. We show that full-information optimization admits a PTAS but no EPTAS under Gap-ETH, and that the objective is monotone but not submodular. This motivates a weighted ordinal coverage relaxation, which is submodular and supports a failure-conditioned greedy oracle under pairwise feedback. We then convert this oracle back into $θ$-type guarantees through finite-family auditing or a minimax wrapper. We also provide small-scale LLM experiments illustrating the predicted query savings and the role of complementarity in committee selection.

模型集成反馈学习高效评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。