用视觉语言模型识别关键比较，让人工只做必要判断，大幅节省标注成本。
Surprise-Guided MergeSort: Budget-Efficient Human-in-the-Loop Ranking via Adaptive Comparison Scheduling
- 通过自下而上的归并排序结构，利用传递性减少重复比较。
- 每轮可跳过最多535个无信息量的比较，相同预算下准确率提升6%~12%。
- 适合需要高效人工参与排序的多模态评估任务，如图像质量、文本相似度。
成对比较是主观排序任务的金标准，但全量标注需 $O(n^2)$ 次人工判断。尽管基于排序的方法已将复杂度降至 $O(n/log n)$，仍需人工参与每一项比较。为提升效率，我们提出将视觉语言模型（VLM）作为问题优先级处理器，而非替代者。所提的 **Surprise-Guided MergeSort (SGS)** 框架包含三部分：(1) 自底向上的归并排序调度器，利用传递性组织比较；(2) 复合惊喜评分器——结合去位置偏置的VLM置信度、Elo差距与投票熵，量化比较模糊度；(3) 自适应预算分配器，将高惊喜对交给人类，低惊喜对通过传递性自动推断。在涵盖文本相似度（STS-B、BIOSSES、SICKR-STS）和图像质量评估（KonIQ-10k、TID2013、LIVE Challenge）的六个基准上验证，SGS每会话可识别并跳过最多535个非信息性比较，相同总预算下，其肯德尔 τ×100 提升6至12个百分点，显著优于Active Elo。结果表明，结合VLM引导的惊喜度量与算法排序，可在多领域实现稳定且高效的精度-效率权衡。
原文摘要 · Abstract (English)
Pairwise comparison is the gold standard for subjective ranking tasks; however, exhaustive annotation requires a massive number of human comparisons ($O(n^2)$). While sorting-based methods have reduced this burden to $O(n\log n)$, they still require expensive human judgment for every single comparison. To further improve annotation efficiency, we propose leveraging a Vision-Language Model (VLM) not as an annotator replacement, but as a \emph{question prioritizer} to identify which comparisons genuinely require human judgment. The proposed \textbf{Surprise-Guided MergeSort (SGS)} framework achieves this through three integrated components: (1) a bottom-up MergeSort scheduler that structures comparisons and exploits transitivity, (2) a composite Surprise Scorer -- combining position-bias-cancelled VLM confidence, Elo gap, and vote entropy -- to quantify comparison ambiguity, and (3) an adaptive budget allocator that routes high-surprise pairs to humans while automating low-surprise pairs via transitivity inference. Validation was conducted on six diverse benchmarks spanning text similarity (STS-B, BIOSSES, SICKR-STS) and image quality assessment (KonIQ-10k, TID2013, LIVE Challenge). SGS effectively identified and skipped up to 535 non-informative comparisons per session. Consequently, it achieved Kendall's $τ{\times}100$ improvements of $+6$ to $+12$ over Active Elo under the same total budget. These results demonstrate that combining VLM-guided surprise metrics with algorithmic sorting provides a generally consistent accuracy-efficiency trade-off across diverse domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。