轻量模型在极少标注数据下实现高效文本分类
Lightweight Contenders: Navigating Semi-Supervised Text Mining through Peer Collaboration and Self Transcendence
- 通过师生协同与同伴互训提升小模型性能
- 在极少量标注数据下超越FLiText和DisCo等先进框架
- 适合资源受限场景下的半监督文本挖掘
轻量级模型在半监督学习（SSL）中需减少标注样本并实现低成本推理，但训练标签稀缺导致参数受限，制约了SSL表现。本文提出PS-NET框架，专为轻量级模型的半监督文本挖掘设计。该框架采用在线知识蒸馏，让轻量学生模型模仿教师模型；引入学生同伴集成机制，实现相互指导；并实施持续对抗扰动策略，促进渐进式泛化自增强。实验表明，配备两层蒸馏BERT的PS-NET，在极少数标注数据条件下，于半监督文本分类任务中显著优于当前最先进轻量级SSL框架FLiText与DisCo。
原文摘要 · Abstract (English)
The semi-supervised learning (SSL) strategy in lightweight models requires reducing annotated samples and facilitating cost-effective inference. However, the constraint on model parameters, imposed by the scarcity of training labels, limits the SSL performance. In this paper, we introduce PS-NET, a novel framework tailored for semi-supervised text mining with lightweight models. PS-NET incorporates online distillation to train lightweight student models by imitating the Teacher model. It also integrates an ensemble of student peers that collaboratively instruct each other. Additionally, PS-NET implements a constant adversarial perturbation schema to further self-augmentation by progressive generalizing. Our PS-NET, equipped with a 2-layer distilled BERT, exhibits notable performance enhancements over SOTA lightweight SSL frameworks of FLiText and DisCo in SSL text classification with extremely rare labelled data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。