用CLIP快速预判多模态模型标注难易度，省时省力。
AnchorScore: A CLIP-Based Diagnostic of MLLM Annotation Difficulty
- 基于CLIP的锚点评分（AnchorScore）评估每类标注难度。
- 在13个课堂行为类别中与实际精度相关性达0.769，显著优于其他方法。
- 适合需优化人工审核或减少大模型调用成本的场景。
多模态大语言模型（MLLM）广泛用于自动化标注，但其各分类准确率差异大（如三个教室子数据集的13个类别间为12%–98%），且评估成本高：运行一个270亿参数的MLLM对5,416张验证图像需约14小时，而冻结版CLIP仅需约3分钟。目前缺乏低成本的先验难度排序信号。本文基于前序研究提出的锚点代理（每类零样本CLIP准确率）构建全图版本的AnchorScore，作为预判MLLM标注可靠性低的诊断工具。在课堂行为数据（SCB5，13类，6个MLLM）上，AnchorScore与各类别实际准确率的相关性达斯皮尔曼等级相关系数0.769（p=0.002，n=13）。其他难度预测器（DINOv2、ResNet-50、SigLIP、MLLM自述不确定性）在n=13时均未表现出显著类级相关性。跨模型一致性控制表明，AnchorScore主要捕捉共享的类别难度因子而非CLIP特有信号。在Stanford40 Actions独立复现中相关性为0.817（p<0.001）；该关联在动作识别数据上最强，医疗与卫星影像上减弱。三个应用包括：可部署的混合路由策略（预测类路由使性能提升最高达+23个百分点，同时节省约44%的MLLM计算成本）、困难类别提示消歧（探索性）、人工审核优先级预测。AnchorScore不估计精确准确率，而是提供低成本排名信号，引导昂贵的MLLM评估聚焦于信息量最大的类别。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) are widely used for automated annotation, yet their per-class accuracy varies widely (e.g., 12%-98% across the 13 classes of three classroom sub-datasets) and is expensive to measure: evaluating one 27B MLLM on 5,416 validation images takes roughly 14 hours, whereas a frozen-CLIP pass over the same images completes in about 3 minutes. A low-cost signal for ranking classes by expected MLLM annotation difficulty a priori remains underexplored. Building on the AnchorProxy construct (per-class zero-shot CLIP accuracy) introduced in the companion study, this paper systematically evaluates its full-frame formulation, termed AnchorScore here, as an a priori diagnostic that flags the classes MLLMs are least likely to annotate reliably. On classroom behavior data (SCB5, 13 classes, 6 MLLMs), AnchorScore correlates with per-class MLLM accuracy (Spearman rho = 0.769, p = 0.002, n = 13). None of the alternative difficulty predictors (DINOv2, ResNet-50, SigLIP, or MLLM self-verbalized uncertainty) showed a significant class-level correlation at n = 13. A cross-model consensus control suggests AnchorScore primarily captures a shared class-difficulty factor rather than a CLIP-specific signal. An independent replication on Stanford40 Actions yields a nearly identical effect (rho = 0.817, p < 0.001); the association is strongest on activity-recognition data and attenuates on medical and satellite imagery. Three practical applications follow: a deployable hybrid CLIP/MLLM routing strategy (predicted-class routing: up to +23 pp over CLIP-only at roughly 44% MLLM cost savings), prompt disambiguation on hard classes (exploratory), and review-priority prediction for human verification. AnchorScore does not estimate exact MLLM accuracy; it provides a low-cost ranking signal that directs expensive MLLM evaluation to the classes where it is most informative.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。