用大模型生成标签，低成本高质量标注产品使用场景
Learning to Predict Usage Options of Product Reviews with LLM-Generated Labels
- 用大模型少样本学习生成标注数据，训练专用预测模型
- 自研模型标注质量超第三方服务，GPT-4标签媲美领域专家
- 适合需要隐私控制和能效优化的工业级文本标注场景
标注大规模数据集具有挑战性，而众包常成本高且质量不稳定，尤其在复杂自然语言任务中。本文提出一种利用大模型作为少样本学习者的方法，为产品评论中预测使用场景的任务生成标注数据，并训练一个独立模型进行预测。同时提出新评估指标HAMS4，用于比较一组输出与多参考集的匹配度。相比直接使用大模型进行序列到序列任务，自研模型可更好控制能耗与隐私。实验表明，该方法在成本上显著优于传统方式，生成数据质量超过第三方服务，且GPT-4生成的标签达到领域专家水平。代码与生成标签已公开。
原文摘要 · Abstract (English)
Annotating large datasets can be challenging. However, crowd-sourcing is often expensive and can lack quality, especially for non-trivial tasks. We propose a method of using LLMs as few-shot learners for annotating data in a complex natural language task where we learn a standalone model to predict usage options for products from customer reviews. We also propose a new evaluation metric for this scenario, HAMS4, that can be used to compare a set of strings with multiple reference sets. Learning a custom model offers individual control over energy efficiency and privacy measures compared to using the LLM directly for the sequence-to-sequence task. We compare this data annotation approach with other traditional methods and demonstrate how LLMs can enable considerable cost savings. We find that the quality of the resulting data exceeds the level attained by third-party vendor services and that GPT-4-generated labels even reach the level of domain experts. We make the code and generated labels publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。