用CLIP预排序+人机协作，大幅降低主观评分的标注成本。
EZ-Sort: Efficient Pairwise Comparison via Zero-Shot CLIP-Based Pre-Ordering and Human-in-the-Loop Sorting
- 用CLIP零样本预排序，减少人工比较次数
- 相比完全人工标注节省90.5%工作量（n=100）
- 适合图像质量评估等主观打分场景
成对比较在主观或复杂标注任务中因可靠性高而被青睐，但全量比较需O(n²)标注。现有方法通过主动采样将负担降至O(n log n)。本文提出EZ-Sort：首先利用无训练的CLIP模型进行层级化零样本预排序，再初始化带桶感知的Elo分数，最后运行基于不确定性的自适应人机协同归并排序。在人脸年龄估计（FGNET）、历史图像年代排序（DHCI）和视网膜图像质量评估（EyePACS）上验证，当n=100时，相比全量比较减少90.5%人工成本，比先前方法降低19.8%，同时保持或提升评分者间一致性。结果表明，结合CLIP先验与不确定性引导采样，可实现高效可扩展的成对排序。
原文摘要 · Abstract (English)
Pairwise comparison is often favored over absolute rating or ordinal classification in subjective or difficult annotation tasks due to its improved reliability. However, exhaustive comparisons require a massive number of annotations (O(n^2)). Recent work has greatly reduced the annotation burden (O(n log n)) by actively sampling pairwise comparisons using a sorting algorithm. We further improve annotation efficiency by (1) roughly pre-ordering items using the Contrastive Language-Image Pre-training (CLIP) model hierarchically without training, and (2) replacing easy, obvious human comparisons with automated comparisons. The proposed EZ-Sort first produces a CLIP-based zero-shot pre-ordering, then initializes bucket-aware Elo scores, and finally runs an uncertainty-guided human-in-the-loop MergeSort. Validation was conducted using various datasets: face-age estimation (FGNET), historical image chronology (DHCI), and retinal image quality assessment (EyePACS). It showed that EZ-Sort reduced human annotation cost by 90.5% compared to exhaustive pairwise comparisons and by 19.8% compared to prior work (when n = 100), while improving or maintaining inter-rater reliability. These results demonstrate that combining CLIP-based priors with uncertainty-aware sampling yields an efficient and scalable solution for pairwise ranking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。