用合成数据修复模型因序列化等导致的性能下降
Recover-LoRA: Data-Free Accuracy Recovery of Degraded Language Models via Low-Rank Adaptation
- 通过合成数据与对齐训练，仅在部分层上微调低秩适配器
- 在小模型上恢复5%-17%的准确率，适用于MHA和GQA架构
- 无需真实数据，适合部署后模型修复场景
推理优化如量化、剪枝、格式转换、模型导出和序列化可能导致语言模型任务性能下降。现有研究多关注鲁棒量化，而本文聚焦于从各类权重退化源头（如错误序列化）中恢复模型精度。提出 Recover-LoRA：一种轻量级、无需数据的方法，通过合成数据与逻辑值蒸馏，在选定层上学习 LoRA 适配器，使退化模型与全精度模型对齐。我们在多种小型语言模型（包括 MHA 与 GQA 架构）上验证其有效性，结果表明在 MHA 和 GQA 小模型上可恢复 5%-17% 的准确率。
原文摘要 · Abstract (English)
Inference optimizations such as quantization, pruning, format and datatype conversion, model export, and serialization can lead to functional degradations in language model task performance. While most efforts on performance recovery for deployment focus on robust quantization techniques, we focus on recovering model accuracies from any sources that degrade model weights, such as improper model serialization. In this work, we propose Recover-LoRA, a lightweight and dataset agnostic method to recover accuracy in degraded models. Recover-LoRA uses synthetic data and logit distillation to learn LoRA adapters on selective layers that facilitate aligning the degraded model to its full precision model. We investigate the utility of Recover-LoRA across a diverse set of small language models (SLMs), including models with varying attention architectures, multi-head attention (MHA) and group-query attention (GQA), as well as several evaluation datasets. Our results show that Recover-LoRA recovers model accuracies by 5-17% on MHA and GQA SLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。