用智能路由降低标注成本，同时保证误差可控
HyPAC: Cost-Efficient LLMs-Human Hybrid Annotation with PAC Error Guarantees
- 根据不确定度动态分配数据给最省钱的标注源
- 实验显示成本降低78.51%且误差严格受控
- 无需依赖数据分布或预训练模型，适合高性价比标注场景
数据标注常需在不同来源间权衡成本与质量，如快速的大语言模型（LLMs）、慢速的推理模型和人工专家。本文研究如何将输入路由至最经济的标注源，同时控制测试样本的标注误差。提出HyPAC方法，通过重要性采样和置信上界校准两个决策阈值，依据不确定性将输入分为三类区域，并分配至合适的标注源。理论上证明，HyPAC在无分布假设和无需预训练模型的前提下，实现最小期望成本，并提供分布无关的大概率近似正确（PAC）误差保证。在多个基准数据集上的实验表明，该方法可降低78.51%的标注成本，同时精准控制标注误差。
原文摘要 · Abstract (English)
Data annotation often involves multiple sources with different cost-quality trade-offs, such as fast large language models (LLMs), slow reasoning models, and human experts. In this work, we study the problem of routing inputs to the most cost-efficient annotation source while controlling the labeling error on test instances. We propose \textbf{HyPAC}, a method that adaptively labels inputs to the most cost-efficient annotation source while providing distribution-free guarantees on annotation error. HyPAC calibrates two decision thresholds using importance sampling and upper confidence bounds, partitioning inputs into three regions based on uncertainty and routing each to the appropriate annotation source. We prove that HyPAC achieves the minimum expected cost with a probably approximately correct (PAC) guarantee on the annotation error, free of data distribution and pre-trained models. Experiments on common benchmarks demonstrate the effectiveness of our method, reducing the annotation cost by 78.51\% while tightly controlling the annotation error.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。