PARA通过动态调整隐藏表示，实现高效高精度微调。
PARA: Parameter-Efficient Fine-tuning with Prompt Aware Representation Adjustment
- 在每个Transformer层加入轻量级提示感知向量生成器
- 性能超越现有参数高效微调方法，参数量相当
- 单模型多租户场景下比LoRA更高效，适合工业部署
在参数高效微调（PEFT）领域，尽管已有LoRA等方法，但产业界仍亟需一种在单骨干多租户应用中兼具高效与高性能的PEFT方案。本文提出一种新颖且简洁的PEFT技术——提示感知表示调整（PARA）。其核心是在每个Transformer层内集成一个轻量级向量生成器，该生成器根据输入提示动态生成响应向量，从而调整隐藏表示。我们在多种任务上的大量实验表明：首先，PARA在可调参数数量相当的情况下，性能超越当前主流PEFT基准；其次，在单骨干多租户场景中，其效率优于LoRA，展现出显著的工业应用潜力。
原文摘要 · Abstract (English)
In the realm of parameter-efficient fine-tuning (PEFT) methods, while options like LoRA are available, there is a persistent demand in the industry for a PEFT approach that excels in both efficiency and performance within the context of single-backbone multi-tenant applications. This paper introduces a new and straightforward PEFT technique, termed \underline{P}rompt \underline{A}ware \underline{R}epresentation \underline{A}djustment (PARA). The core of our proposal is to integrate a lightweight vector generator within each Transformer layer. This generator produces vectors that are responsive to input prompts, thereby adjusting the hidden representations accordingly. Our extensive experimentation across diverse tasks has yielded promising results. Firstly, the PARA method has been shown to surpass current PEFT benchmarks in terms of performance, despite having a similar number of adjustable parameters. Secondly, it has proven to be more efficient than LoRA in the single-backbone multi-tenant scenario, highlighting its significant potential for industrial adoption.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。