用开源大模型自动标注数据，动态融合标签描述提升准确率
Automatic Labelling with Open-source LLMs using Dynamic Label Schema Integration
- 逐个测试标签描述，动态选择最匹配的标签
- 在高基数任务上相比直接分类提升显著性能
- 可灵活平衡标注质量和覆盖范围，适合内部数据标注
获取标注训练数据在真实机器学习项目中仍是一项成本高昂的任务，难以满足数量和质量要求。近期大型语言模型（LLMs），尤其是GPT-4，展现出以高精度标注数据的巨大潜力。然而，隐私和成本问题限制了GPT-4的广泛应用。本文探索如何有效利用开源模型实现自动标注。我们发现引入标签结构是关键方向，但直接使用标签描述进行分类在高基数任务上表现不佳。为此，我们提出检索增强分类（RAC）：让LLM逐个对单个标签进行推理，从最相关的标签开始迭代，直到选出一个被接受的标签。实验表明，该方法通过动态整合标签描述，显著提升了标注任务的性能。此外，通过仅聚焦最有希望的标签，RAC可在标签质量与覆盖范围之间灵活权衡，我们据此成功自动标注了内部数据集。
原文摘要 · Abstract (English)
Acquiring labelled training data remains a costly task in real world machine learning projects to meet quantity and quality requirements. Recently Large Language Models (LLMs), notably GPT-4, have shown great promises in labelling data with high accuracy. However, privacy and cost concerns prevent the ubiquitous use of GPT-4. In this work, we explore effectively leveraging open-source models for automatic labelling. We identify integrating label schema as a promising technology but found that naively using the label description for classification leads to poor performance on high cardinality tasks. To address this, we propose Retrieval Augmented Classification (RAC) for which LLM performs inferences for one label at a time using corresponding label schema; we start with the most related label and iterates until a label is chosen by the LLM. We show that our method, which dynamically integrates label description, leads to performance improvements in labelling tasks. We further show that by focusing only on the most promising labels, RAC can trade off between label quality and coverage - a property we leverage to automatically label our internal datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。