一种高效适配大模型的新方法，支持多任务批处理与可解释性。
3-in-1: 2D Rotary Adaptation for Efficient Finetuning, Efficient Batching and Composability
- 用二维旋转实现参数高效微调，仅需不到0.1%可训练参数
- 单批次内支持不同适配器请求，开销接近逐元素乘法
- 提升模型可解释性，适用于需要组合干预的场景
参数高效微调（PEFT）方法能有效适配大语言模型（LLM）以应对多样下游任务，降低存储和显存需求。然而，一些应用带来了新挑战：当同一批次中不同请求需调用不同任务或用户专用适配器时，如何高效部署成为难题；同时，理解模型运作机制对可解释性至关重要。本文提出新方法RoAd，采用简单二维旋转实现模型适配，解决上述问题：（1）RoAd极具参数效率，在GLUE、8个常识推理任务及4个算术推理任务上表现优异，可训练参数少于0.1%；（2）支持单批次内混合使用不同适配器，计算开销接近逐元素乘法而非批量矩阵乘法；（3）通过分布式可交换干预框架增强模型可解释性，组合实验验证其有效性。
原文摘要 · Abstract (English)
Parameter-efficient finetuning (PEFT) methods effectively adapt large language models (LLMs) to diverse downstream tasks, reducing storage and GPU memory demands. Despite these advantages, several applications pose new challenges to PEFT beyond mere parameter efficiency. One notable challenge involves the efficient deployment of LLMs equipped with multiple task- or user-specific adapters, particularly when different adapters are needed for distinct requests within the same batch. Another challenge is the interpretability of LLMs, which is crucial for understanding how LLMs function. Previous studies introduced various approaches to address different challenges. In this paper, we introduce a novel method, RoAd, which employs a straightforward 2D rotation to adapt LLMs and addresses all the above challenges: (1) RoAd is remarkably parameter-efficient, delivering optimal performance on GLUE, eight commonsense reasoning tasks and four arithmetic reasoning tasks with $<0.1\%$ trainable parameters; (2) RoAd facilitates the efficient serving of requests requiring different adapters within a batch, with an overhead comparable to element-wise multiplication instead of batch matrix multiplication; (3) RoAd enhances LLM's interpretability through integration within a framework of distributed interchange intervention, demonstrated via composition experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。