用少量标签训练轻量模型，实现高效文本挖掘。
NanoNet: Parameter-Efficient Learning with Label-Scarce Supervision for Lightweight Text Mining Model
- 通过在线知识蒸馏生成多个小模型并互学提升性能。
- 仅需极少标注数据，训练成本低且推理速度快。
- 适合资源受限场景下的快速部署与迭代。
轻量半监督学习（LSL）策略在节约标注样本和降低模型推理开销方面表现出色。现有研究虽有效应用了从大模型到小模型的知识迁移与协同训练正则化，但这类方法计算成本高且易陷入局部最优，难以找到理想解。为此，我们探索了三种低成本文本挖掘场景的可行性：有限标注监督、轻量微调及快速推理的小模型。提出NanoNet框架，一种基于参数高效学习的轻量文本挖掘方法。该框架采用在线知识蒸馏生成多个小模型，并通过相互学习正则化增强其性能。整个过程利用参数高效学习机制，显著降低训练成本与标注需求，最终获得适用于下游推理的轻量模型。
原文摘要 · Abstract (English)
The lightweight semi-supervised learning (LSL) strategy provides an effective approach of conserving labeled samples and minimizing model inference costs. Prior research has effectively applied knowledge transfer learning and co-training regularization from large to small models in LSL. However, such training strategies are computationally intensive and prone to local optima, thereby increasing the difficulty of finding the optimal solution. This has prompted us to investigate the feasibility of integrating three low-cost scenarios for text mining tasks: limited labeled supervision, lightweight fine-tuning, and rapid-inference small models. We propose NanoNet, a novel framework for lightweight text mining that implements parameter-efficient learning with limited supervision. It employs online knowledge distillation to generate multiple small models and enhances their performance through mutual learning regularization. The entire process leverages parameter-efficient learning, reducing training costs and minimizing supervision requirements, ultimately yielding a lightweight model for downstream inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。