用2维提示嵌入实现98%参数压缩，性能不降反升。
Ultra-Low-Dimensional Prompt Tuning via Random Projection
- 在极低维空间优化提示，通过冻结随机矩阵上投影
- 相比原始提示调优减少98%训练参数，保持性能不变
- 适合大规模语言模型轻量定制，存储效率极高
大型语言模型虽性能卓越，但微调成本日益高昂。提示调优是一种参数高效的微调方法，通过学习提示嵌入来减少参数量，但这些嵌入通常与模型隐藏维度绑定，限制了参数节省。本文提出超低维提示调优（ULPT），在低维空间（如2维）优化提示，并使用冻结的随机矩阵进行上投影。实验表明，ULPT相比原始提示调优可减少98%的训练参数，同时保持性能。我们在超过20个NLP任务上验证了该方法，结果一致优于近期参数高效微调方法，且使用参数显著更少，是大规模语言模型定制的理想存储高效框架。
原文摘要 · Abstract (English)
Large language models achieve state-of-the-art performance but are increasingly costly to fine-tune. Prompt tuning is a parameter-efficient fine-tuning method that addresses parameter-efficiency by learning prompt embeddings, but these embeddings are typically tied to the model's hidden dimensionality, limiting parameter saving. In this paper, we propose Ultra-Low-dimensional Prompt Tuning (ULPT), a simple yet effective method that optimizes prompts in a low-dimensional space (e.g., 2D) and uses a frozen random matrix for up-projection. ULPT can achieve 98% reduction in the training parameters compared to vanilla prompt tuning while preserving performance. Our extensive experiments across over 20 NLP tasks demonstrate that ULPT consistently outperforms recent parameter-efficient tuning methods using significantly fewer parameters, making it well-suited as a storage-efficient framework for massive LLM customization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。