用知识图谱指导大模型团队纠错,提升工业级数据标注精度
Adjudicator: Correcting Noisy Labels with a KG-Informed Council of LLM Agents
- 构建动态知识图谱统一物品上下文,驱动多智能体辩论投票
- 在1000项数据上达0.99 F1,精准识别复杂结构性错误
- 适合需要高可信数据的工业场景,结果可解释且可部署
生产机器学习系统的性能受限于训练数据质量。在高风险工业应用中,标签噪声会降低模型表现并损害用户信任。本文提出Adjudicator系统,解决自动识别与修正标签噪声这一关键数据挖掘挑战,并已通过生产环境验证。该系统将问题建模为神经符号任务:首先构建动态知识图谱(KG)统一项目上下文;随后基于此构建“智能体委员会”——一种新型多智能体大语言模型架构,让专业化智能体辩论并投票判断标签有效性。我们在AlleNoise基准的1000项平衡子集上验证系统,基于KG的模型取得0.99 F1分数,显著优于单个大模型基线(0.48 F1)和无KG的委员会(0.59 F1)。分析表明,其优势源于新颖的覆盖逻辑:利用KG实现完美召回(Complete Recall),精准识别复杂结构性错误,而基线方法无法发现此类错误。该结果展示了自动化、高精度且可解释的数据验证系统,为严格监管环境下生成黄金数据集提供了有力证明。
原文摘要 · Abstract (English)
The performance of production machine learning systems is fundamentally limited by the quality of their training data. In high-stakes industrial applications, noisy labels can degrade performance and erode user trust. This paper presents Adjudicator, a system that addresses the critical data mining challenge of automatically identifying and correcting label noise and has been validated for production deployment. Adjudicator models this as a neuro-symbolic task, first constructing a dynamic Knowledge Graph (KG) to unify item context. This KG then informs a "Council of Agents," a novel multi-agent Large Language Model architecture where specialized agents debate and vote on a label's validity. We validate our system on a 1,000-item balanced subset of the AlleNoise benchmark. Our KG-informed model achieves a 0.99 F1-score, significantly outperforming a single-LLM baseline (0.48 F1) and a non-KG council (0.59 F1). Our analysis reveals this is due to a Precision, achieved by a novel override logic that uses the KG to perfectly identify complex, structural errors (complete Recall) -- a class of errors that baselines fail to find. This result demonstrates a robust and explainable system for automated, high-precision data verification, serving as a vital proof-of-concept for generating golden datasets in strictly governed industrial environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。