跨任务提示调优让大模型在多个任务间高效共享知识
CrossPT: Exploring Cross-Task Transferability through Multi-Task Prompt Tuning
- 将提示分为共享与任务专属两部分，用注意力机制融合
- 在低资源场景下准确率显著高于传统方法
- 适合需要多任务学习且参数受限的研究者
提示调优为适配大预训练语言模型到新任务提供了一种参数高效的方法，但现有方法大多针对单任务设计，无法在相关任务间共享知识。我们提出跨任务提示调优（CrossPT），一种模块化多任务提示调优框架，可在保持任务特异性的同时实现可控的知识迁移。CrossPT将每个目标提示分解为共享的预训练源提示和任务专属私有提示，并通过可学习的注意力机制组合。为支持稳健迁移，我们系统研究了提示初始化、共享与私有提示平衡、源提示数量、学习率、任务前缀及标签语义等关键设计因素。在GLUE及相关基准上的实证结果表明，相比传统提示调优和相关方法，CrossPT在低资源场景下实现了更高的准确率与鲁棒性，同时保持强参数效率。
原文摘要 · Abstract (English)
Prompt tuning offers a parameter-efficient way to adapt large pre-trained language models to new tasks, but most existing approaches are designed for single-task settings, failing to share knowledge across related tasks. We propose Cross-task Prompt Tuning (CrossPT), a modular framework for multi-task prompt tuning that enables controlled knowledge transfer while maintaining task-specific specialization. CrossPT decomposes each target prompt into shared, pre-trained source prompts and task-specific private prompts, combined via a learned attention mechanism. To support robust transfer, we systematically investigate key design factors including prompt initialization, balancing shared and private prompts, number of source prompts, learning rates, task prefixes, and label semantics. Empirical results on GLUE and related benchmarks show that CrossPT achieves higher accuracy and robustness compared to traditional prompt tuning and related methods, particularly in low-resource scenarios, while maintaining strong parameter efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。