用轻量适配器提升压缩模型性能，让小模型更接近大模型表现。
Alignment Adapter to Improve the Performance of Compressed Deep Learning Models
- 设计滑动窗口适配器，对齐压缩模型与原大模型的词元嵌入
- 在三个NLP任务上使压缩模型性能显著提升，开销极小
- 可即插即用或联合微调，适配任意压缩方法
压缩深度学习模型对于资源受限环境下的部署至关重要，但其性能常落后于大规模模型。为弥合这一差距，我们提出轻量级、基于滑动窗口的对齐适配器（Alignment Adapter, AlAd），用于对齐压缩模型与原始大模型之间的词元级嵌入。AlAd保留局部上下文语义，支持不同维度或架构间的灵活对齐，且完全不依赖具体压缩方法。AlAd可作为冻结压缩模型的即插即用模块部署，也可与压缩模型联合微调以进一步提升性能。在BERT系列模型上针对三个词元级自然语言处理任务的实验表明，AlAd在仅引入极小的模型尺寸和延迟开销下，显著提升了压缩模型的性能。
原文摘要 · Abstract (English)
Compressed Deep Learning (DL) models are essential for deployment in resource-constrained environments. But their performance often lags behind their large-scale counterparts. To bridge this gap, we propose Alignment Adapter (AlAd): a lightweight, sliding-window-based adapter. It aligns the token-level embeddings of a compressed model with those of the original large model. AlAd preserves local contextual semantics, enables flexible alignment across differing dimensionalities or architectures, and is entirely agnostic to the underlying compression method. AlAd can be deployed in two ways: as a plug-and-play module over a frozen compressed model, or by jointly fine-tuning AlAd with the compressed model for further performance gains. Through experiments on BERT-family models across three token-level NLP tasks, we demonstrate that AlAd significantly boosts the performance of compressed models with only marginal overhead in size and latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。