用多标注者协作优化数据标注质量，节省成本。
QUORUM: QUality-Optimized Routing Using Multiple annotators
- 根据实例难易度动态分配人类或大模型标注
- 多标注融合通过一致度奖励提升可靠性，质量最高提升34.4%
- 适合需要高质量标注且预算有限的NLP项目
数据标注仍是自然语言处理中的核心瓶颈，需人力投入以获取大规模高质量标签。尽管大语言模型（LLMs）提供了快速低成本的替代方案，但其可靠性高度依赖具体输入：在简单任务上表现良好，但在需要细致推理或上下文理解的任务上常出错。本文提出QUORUM（QUality-Optimized Routing Using Multiple annotators），一个预算感知的动态路由框架，在固定标注预算下，根据实例难度自动分配给人类或LLM标注者，并支持每个实例多次标注。与依赖模型置信度或不确定性估计的方法不同，QUORUM利用基于特征的信号评估实例难度，通过一致性奖励机制融合多标注结果以增强可靠性。我们在英文和多语言环境下的多种封闭式与开放式标注任务中验证了QUORUM，结果表明其标注质量最高提升34.4%，同时成本降低8.8%。代码已开源于https://github.com/amazon-science/QUORUM。
原文摘要 · Abstract (English)
Data annotation remains a central bottleneck in natural language processing, requiring human effort to obtain high-quality labels at scale. While Large Language Models (LLMs) offer a fast and cost-effective alternative, their reliability is highly instance-dependent: they perform well on simple inputs but often fail on examples requiring nuanced reasoning or contextual understanding. In this work, we address this challenge with QUORUM (QUality-Optimized Routing Using Multiple annotators), a budget-aware routing framework that dynamically assigns each instance to human or LLM annotators under a fixed annotation budget. Unlike prior approaches relying on model confidence or uncertainty estimates, QUORUM leverages feature-based signals to estimate instance difficulty and supports multiple annotations per instance, combining them through agreement-based rewards to improve reliability. We evaluate QUORUM across diverse closed- and open-ended annotation tasks in English and multilingual settings, and QUORUM improves annotation quality by up to 34.4% while reducing costs by 8.8% over competing methods. Code can be found at https://github.com/amazon-science/QUORUM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。