用大模型实时检查标注质量，提升数据可靠性。
RE-AD: Real-Time Requirement Adherence for Data Labeling
- 将操作规范拆解为原子规则，分层验证标注
- 合成数据集上F1达0.749，生产中82%错误被修正
- 适合需要高质量标注的LLM训练场景
人工标注数据仍是训练前沿大语言模型的基础。然而，众包标注常因标注者理解偏差或参与度不足导致质量问题。为此，我们提出一种实时要求遵从（RE-AD）框架，利用大模型主动验证标注质量。方法上，通过自我反思将标准操作流程（SOP）分解为原子规则，按复杂度分类，并应用分层验证策略。在合成基准上的评估显示，系统F1得分为0.749。此外，在生产部署中，82%被框架标记的错误被标注者接受并修复。我们还进行了消融实验，验证了核心设计决策的影响。
原文摘要 · Abstract (English)
Human-annotated data remains fundamental to training frontier Large Language Models (LLMs). However, crowd-sourced annotations often suffer from quality issues stemming from annotator misunderstanding or lack of engagement. To address this, we introduce a real-time requirement adherence (RE-AD) framework that leverages LLMs to proactively validate labeling quality. Our methodology involves decomposing Standard Operating Procedures (SOPs) into atomic rules via self-reflection, categorizing them by complexity, and applying tiered validation strategies. Evaluated on a synthetic benchmark, the system achieved an F1 score of 0.749. Furthermore, production deployment resulted in annotators accepting and fixing 82% of the errors flagged by the framework. We include ablation studies to demonstrate the impact of our core design decisions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。