让提示词按任务层级动态组合，提升多任务学习效果
HiVe: Beyond Static Prompts for Multitask Learning via Hierarchy-based Vertical Mixture-of-Experts
- 构建多层级提示结构，根据输入动态选择专用提示
- 在12个任务上超越现有提示调优方法，平均提升5.3%
- 适合需要高效适配多任务的模型微调场景
随着大语言模型持续扩展，参数高效微调（PEFT）成为全参数调整的实用替代方案。提示调优虽有效，但现有方法或采用扁平提示结构，或使用固定组成层级结构，限制了提示的自适应专业化。为解决这一问题，我们提出HiVe，一种提示调优框架，能在训练中利用任务间关系构建提示层次结构，并在推理时采用垂直混合专家（V-MoE）机制，根据输入需求动态组合至所需专业层级的提示。实验表明，HiVe在多个任务上持续优于强基线提示调优方法。
原文摘要 · Abstract (English)
As large language models (LLMs) continue to scale, parameter-efficient fine-tuning (PEFT) has become a practical alternative to full-parameter adaptation. Prompt tuning is effective, but existing approaches either use flat prompt structures or hierarchical structures with fixed prompt composition, limiting adaptive prompt specialization. To address this limitation, we propose HiVe, a prompt tuning framework that models prompts at multiple levels and enables input-dependent specialization. HiVe constructs a prompt hierarchy by leveraging inter-task relationships during training, and employs a vertical mixture-of-experts (V-MoE) mechanism at inference time to compose prompts up to the level of specialization required for each input. Experiments show that HiVe consistently outperforms strong prompt tuning baselines across diverse tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。