用有意义的词替代随机标记，让大模型微调更高效
Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuning
- 用语义明确的词替换传统随机标记，提升提示词设计合理性
- 训练速度更快，参数量更少，在文本分类等任务上表现更优
- 适合追求高效微调的开发者，尤其关注资源受限场景
近年来，大语言模型（LLMs）因提示工程成本低而在特定任务中广泛应用。标准方法如前缀微调使用无语义意义的可变标记，需大量训练才能达到最佳性能，常表现不足。本文提出一种新方法——语义知识微调（SK-Tuning），在提示和前缀微调中采用具有实际语义的词汇，而非随机标记。该方法利用固定预训练模型通过零样本能力理解并处理提示的语义内容，再将处理后的提示与输入文本融合，以提升模型在特定任务上的表现。实验结果表明，相较于其他微调方法，SK-Tuning具备更快的训练速度、更少的参数量，并在文本分类与理解等任务上取得更优性能。该方法为提升大模型在语言任务中的效率与效果提供了有效路径。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are gaining significant popularity in recent years for specialized tasks using prompts due to their low computational cost. Standard methods like prefix tuning utilize special, modifiable tokens that lack semantic meaning and require extensive training for best performance, often falling short. In this context, we propose a novel method called Semantic Knowledge Tuning (SK-Tuning) for prompt and prefix tuning that employs meaningful words instead of random tokens. This method involves using a fixed LLM to understand and process the semantic content of the prompt through zero-shot capabilities. Following this, it integrates the processed prompt with the input text to improve the model's performance on particular tasks. Our experimental results show that SK-Tuning exhibits faster training times, fewer parameters, and superior performance on tasks such as text classification and understanding compared to other tuning methods. This approach offers a promising method for optimizing the efficiency and effectiveness of LLMs in processing language tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。