用扩散模型生成高质量提示，提升少样本策略泛化能力。
Prompt Tuning with Diffusion for Few-Shot Pre-trained Policy Generalization
- 将提示调优转化为条件生成任务，从随机噪声生成提示。
- 在元强化学习任务中实现优于基线的少样本泛化性能。
- 适合需要快速适应新任务的预训练策略模型研究者。
离线强化学习方法利用历史经验构建最优策略，构成预训练大规模模型（PLMs）的基础。面对未见任务时，PLMs常通过少量专家轨迹作为提示来加速适应。尽管已有多种提示调优方法提升提示质量，但其优化常受限于提示初始化，显著压缩探索空间，可能导致次优解。为摆脱初始提示依赖，本文转向生成模型视角，将提示调优视为条件生成建模，从随机噪声中生成提示。提出的新方法Prompt Diffuser采用条件扩散模型生成高质量提示。框架核心在于轨迹重建机制与训练阶段下游任务引导的精细融合。实验结果表明，Prompt Diffuser在元强化学习任务中表现强劲，是高效可靠的提示调优工具。
原文摘要 · Abstract (English)
Offline reinforcement learning (RL) methods harness previous experiences to derive an optimal policy, forming the foundation for pre-trained large-scale models (PLMs). When encountering tasks not seen before, PLMs often utilize several expert trajectories as prompts to expedite their adaptation to new requirements. Though a range of prompt-tuning methods have been proposed to enhance the quality of prompts, these methods often face optimization restrictions due to prompt initialization, which can significantly constrain the exploration domain and potentially lead to suboptimal solutions. To eliminate the reliance on the initial prompt, we shift our perspective towards the generative model, framing the prompt-tuning process as a form of conditional generative modeling, where prompts are generated from random noise. Our innovation, the Prompt Diffuser, leverages a conditional diffusion model to produce prompts of exceptional quality. Central to our framework is the approach to trajectory reconstruction and the meticulous integration of downstream task guidance during the training phase. Further experimental results underscore the potency of the Prompt Diffuser as a robust and effective tool for the prompt-tuning process, demonstrating strong performance in the meta-RL tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。