用大模型标注可取代人工,成本降为十分之一且效果更优。
Do We Still Need Humans in the Loop? Human vs. LLM Annotation in Active Learning for TikTok Hate Speech Detection
- 采用双问题分解界面,让大模型标注效果接近人类。
- 大模型标注在27.8万条德语政治评论上表现优于人工标注,成本仅十分之一。
- 适合需要低成本高质量标注的团队,尤其关注界面设计与模型选择。
监督学习中的数据标注仍是瓶颈。主动学习(AL)通过筛选信息量高的样本减少人工标注数量;而指令微调的大模型则从另一方向突破,使标注成本极低,可对全语料库进行标注。本文研究两个问题:大模型标注能否替代人工参与主动学习循环?当全语料库可低成本标注时,主动学习是否仍有意义?我们在一个包含27.8万条德语政治类TikTok评论的新数据集上,对比了人工与大模型标注在相同条件下的表现。结果表明,大规模大模型标注在成本仅为十分之一的情况下,性能超过人工标注,涵盖闭源模型GPT-5.2和开源模型Qwen3.5-122B-A10B,且在软标签评估下仍具鲁棒性。该优势依赖于标注界面设计:仅采用与人工任务一致的两问分解结构才能实现。主动学习在预过滤数据池中未展现出优于随机采样的稳定优势。错误模式取决于模型类型:仅GPT-5.2匹配人工的误报/漏报平衡,其他模型过度标记边境管控与经济竞争相关言论。人类仍不可或缺于评估环节;训练标签的选择转向‘哪个大模型’、‘哪种界面’及‘何种数据池形态’。
原文摘要 · Abstract (English)
Annotating data remains a costly bottleneck for supervised NLP. Active learning (AL) reduces the number of human labels needed by selecting only the most informative instances, while instruction-tuned LLMs attack the same bottleneck from the other side, making labels cheap enough to annotate entire corpora. This raises two questions: can LLM labels replace human labels within the AL loop, and does AL remain necessary when entire corpora can be cheaply labeled? We investigate both by training supervised hate speech classifiers on a new dataset of 278K German political TikTok comments, comparing human and LLM annotation under matched conditions. LLM annotation at scale outperforms human-supervised classifiers at roughly one-tenth the cost, for both a closed-source (GPT-5.2) and an open-weight (Qwen3.5-122B-A10B) LLM, and the advantage is robust under soft-label evaluation. It hinges on the annotation interface: only a two-question decomposition mirroring the human annotation task unlocks it. AL provides no reliable advantage over random sampling in our prefiltered pool. Error structure depends on the LLM: only GPT-5.2 matches the human FP/FN balance, while other variants over-flag border-control and economic-competition discourse. Humans remain essential as evaluators; for training labels, the question shifts to which LLM, which interface, and what shape of pool.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。