轻量级Transformer通过任务优先微调，实现文本分类与命名实体识别的高效统一部署。
Multi-Task Pre-Finetuning of Lightweight Transformer Encoders for Text Classification and NER
- 采用任务优先的LoRA模块，让同一编码器适配多任务。
- 在21个下游任务中，命名实体识别提升0.8%，文本分类提升8.8%。
- 适合移动端部署，兼顾性能与计算效率，适合资源受限场景。
在移动平台部署自然语言处理模型需要兼具多样适应性与低内存、低计算开销的模型。本文研究轻量级BERT类编码器在命名实体识别（NER）和文本分类两大基础任务族上的预微调策略。虽然单任务预微调能提升各自任务表现，但直接多任务预微调会引入冲突优化信号，导致整体性能下降。为此，我们提出一种基于任务优先LoRA模块的简单而有效的多任务预微调框架，实现共享编码器主干与模块化适配器。该方法在保持单任务预微调性能的同时满足实际部署约束。21个下游任务实验表明，命名实体识别平均提升0.8%，文本分类平均提升8.8%，验证了其在移动NLP应用中的有效性。
原文摘要 · Abstract (English)
Deploying natural language processing (NLP) models on mobile platforms requires models that can adapt across diverse applications while remaining efficient in memory and computation. We investigate pre-finetuning strategies to enhance the adaptability of lightweight BERT-like encoders for two fundamental NLP task families: named entity recognition (NER) and text classification. While pre-finetuning improves downstream performance for each task family individually, we find that naïve multi-task pre-finetuning introduces conflicting optimization signals that degrade overall performance. To address this, we propose a simple yet effective multi-task pre-finetuning framework based on task-primary LoRA modules, which enables a single shared encoder backbone with modular adapters. Our approach achieves performance comparable to individual pre-finetuning while meeting practical deployment constraint. Experiments on 21 downstream tasks show average improvements of +0.8% for NER and +8.8% for text classification, demonstrating the effectiveness of our method for versatile mobile NLP applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。