用大模型标注数据训练小模型，可能引发性能下降和不稳定。
Feeding LLM Annotations to BERT Classifiers at Your Own Risk
- 用大模型生成标签微调小模型，存在误差传播风险。
- 相比真实标签，准确率和F1均下降，训练结果更不稳定。
- 适合低风险场景，高风险任务需谨慎使用该流程。
在多种场景中，使用大语言模型（LLM）生成的标签来微调小型编码器模型进行文本分类的做法日益流行。尽管该方法在简单、低风险应用中可能合理，但我们的实证分析揭示了合成数据训练中的经典困境在此设置下的具体表现：与使用真实标签训练的模型相比，不仅准确率和F1分数下降，训练过程也表现出更高的不稳定性及过早性能饱和现象。这些发现对实际应用中该方法的可靠性提出质疑。我们通过误差传播视角解释了上述现象，并提出了熵值过滤和集成学习等实用缓解策略。尽管这些启发式方法提供部分改善，仍无法完全消除从大模型标注中传递非随机错误至小模型的根本风险，因此在高风险文本分类任务中应谨慎采用此工作流。
原文摘要 · Abstract (English)
Using LLM-generated labels to fine-tune smaller encoder-only models for text classification has gained popularity in various settings. While this approach may be justified in simple and low-stakes applications, we conduct empirical analysis to demonstrate how the perennial curse of training on synthetic data manifests itself in this specific setup. Compared to models trained on gold labels, we observe not only the expected performance degradation in accuracy and F1 score, but also increased instability across training runs and premature performance plateaus. These findings cast doubts on the reliability of such approaches in real-world applications. We contextualize the observed phenomena through the lens of error propagation and offer several practical mitigation strategies, including entropy-based filtering and ensemble techniques. Although these heuristics offer partial relief, they do not fully resolve the inherent risks of propagating non-random errors from LLM annotations to smaller classifiers, underscoring the need for caution when applying this workflow in high-stakes text classification tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。