让早退出语言模型在跨域场景下更稳定，无需标注数据
DAdEE: Unsupervised Domain Adaptation in Early Exit PLMs
- 用多层级知识蒸馏+对抗训练，让中间层输出对领域变化不敏感
- 在情感分析等任务上，跨域表现优于现有早退出和域适应方法
- 既提速推理又防遗忘，适合真实场景的低资源部署
预训练语言模型（PLMs）虽具备良好准确性和泛化能力，但因体量大导致推理延迟高。早退出（EE）策略通过在中间层设置分类器让样本提前退出以缓解问题，但其分类器对领域变化敏感，泛化能力差。为此，我们提出无监督域自适应早退出框架（DAdEE），采用多层级知识蒸馏实现域不变表征。DAdEE 在每一层使用基于GAN的对抗适应，缩小源域与目标域间的领域差距。附加的早退出不仅加速推理，还通过减少灾难性遗忘和模式崩溃增强域自适应能力，更适用于真实场景。在情感分析、蕴含分类和自然语言推断等任务上的实验表明，DAdEE 在域偏移条件下持续优于现有早退出方法及多种域自适应方法。匿名源码已公开于 https://github.com/Div290/DAdEE。
原文摘要 · Abstract (English)
Pre-trained Language Models (PLMs) exhibit good accuracy and generalization ability across various tasks using self-supervision, but their large size results in high inference latency. Early Exit (EE) strategies handle the issue by allowing the samples to exit from classifiers attached to the intermediary layers, but they do not generalize well, as exit classifiers can be sensitive to domain changes. To address this, we propose Unsupervised Domain Adaptation in EE framework (DADEE) that employs multi-level adaptation using knowledge distillation. DADEE utilizes GAN-based adversarial adaptation at each layer to achieve domain-invariant representations, reducing the domain gap between the source and target domain across all layers. The attached exits not only speed up inference but also enhance domain adaptation by reducing catastrophic forgetting and mode collapse, making it more suitable for real-world scenarios. Experiments on tasks such as sentiment analysis, entailment classification, and natural language inference demonstrate that DADEE consistently outperforms not only early exit methods but also various domain adaptation methods under domain shift scenarios. The anonymized source code is available at https://github.com/Div290/DAdEE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。