用智能选模型的方法，让大模型标注更省钱又准确。
Cost-aware LLM-based Online Dataset Annotation
- 根据数据上下文动态选少数模型参与投票，省计算资源。
- 在MMLU和IMDB上比全量投票准，成本降低超50%。
- 无需预训练或真实标签，适合实时标注场景。
大语言模型（LLMs）的进展使得低人工干预的数据集标注成为可能。尽管通过多个模型的多数投票可提升标签可靠性、缓解单个模型偏差，但重复调用带来高昂计算成本。本文提出一种新型在线框架——成本感知多数投票（CaMVo），实现高效且精准的LLM标注。CaMVo基于上下文嵌入自适应选择每个数据实例的模型子集，平衡置信度与成本，无需预训练或真实标签。利用基于LinUCB的选择机制与置信度得分的贝叶斯估计器，CaMVo对每个模型的标注准确率下界进行估计，并通过加权多数投票聚合结果。在MMLU和IMDB电影评论数据集上的实证评估表明，CaMVo在保持或超越全量多数投票准确率的同时，显著降低标注成本。该方法为动态标注环境中的低成本高效标注提供了实用且稳健的解决方案。
原文摘要 · Abstract (English)
Recent advances in large language models (LLMs) have enabled automated dataset labeling with minimal human supervision. While majority voting across multiple LLMs can improve label reliability by mitigating individual model biases, it incurs high computational costs due to repeated querying. In this work, we propose a novel online framework, Cost-aware Majority Voting (CaMVo), for efficient and accurate LLM-based dataset annotation. CaMVo adaptively selects a subset of LLMs for each data instance based on contextual embeddings, balancing confidence and cost without requiring pre-training or ground-truth labels. Leveraging a LinUCB-based selection mechanism and a Bayesian estimator over confidence scores, CaMVo estimates a lower bound on labeling accuracy for each LLM and aggregates responses through weighted majority voting. Our empirical evaluation on the MMLU and IMDB Movie Review datasets demonstrates that CaMVo achieves comparable or superior accuracy to full majority voting while significantly reducing labeling costs. This establishes CaMVo as a practical and robust solution for cost-efficient annotation in dynamic labeling environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。