用强化学习选提示，让模型少样本下高效跨任务学习。
Prompt-Tuning Bandits: Enabling Few-Shot Generalization for Efficient Multi-Task Offline RL
- 引入强化学习机制动态选择最优轨迹提示
- 在低数据场景下提升任务表现与样本效率
- 无需微调主模型，适合快速部署的多任务场景
提示（Prompting）已成为将大型预训练Transformer模型适配下游任务的主要范式。提示决策变换器（PDT）通过随机轨迹提示识别目标任务，实现了大规模多任务离线强化学习（RL）预训练。然而，这些提示从专家演示中均匀采样，忽略了关键问题：并非所有提示对区分不同任务都同样有效。这限制了泛化能力与适应性，尤其在低数据或开放世界设置中，样本效率至关重要。为此，我们提出一种轻量级、推理时的基于博弈论的提示调优框架。该框架通过博弈探索并优化轨迹提示选择，以提升任务性能，同时避免对Transformer主干进行代价高昂的微调。实验表明，相比基线方法，基于博弈的提示调优不仅带来明显性能提升，还表现出更优的样本复杂度、可扩展性及提示空间探索能力。结果凸显了自适应提示选择机制在高效离线多任务强化学习中的重要性。
原文摘要 · Abstract (English)
Prompting has emerged as the dominant paradigm for adapting large, pre-trained transformer-based models to downstream tasks. The Prompting Decision Transformer (PDT) enables large-scale, multi-task offline Reinforcement Learning (RL) pre-training by leveraging stochastic trajectory prompts to identify the target task. However, these prompts are sampled uniformly from expert demonstrations, overlooking a critical limitation: not all prompts are equally informative for differentiating between tasks. This limits generalization and adaptation, especially in low-data or open-world settings where sample efficiency is crucial. To address this issue, we propose a lightweight, inference-time, bandit-based prompt-tuning framework. The bandit explores and optimizes trajectory prompt selection to enhance task performance, while avoiding costly fine-tuning of the transformer backbone. Our experiments indicate not only clear performance gains due to bandit-based prompt-tuning, but also better sample complexity, scalability, and prompt space exploration compared to prompt-tuning baselines. These results highlights the importance of adaptive prompt selection mechanisms for efficient generalization in offline multi-task RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。