让预训练策略在不交互环境下快速组合,适应新任务。
Lever: Inference-Time Policy Reuse under Support Constraints

- 用行为嵌入检索并评估策略,离线组合新策略
- 在网格环境中性能媲美甚至超过从零训练
- 适合需要快速部署、无环境交互的场景
强化学习策略通常针对固定目标训练,任务变化后难以复用。本文研究推理时策略复用:给定预训练策略库和新复合目标,能否完全离线构建高质量策略?提出LEVER(Leveraging Efficient Vector Embeddings for Reusable policies)框架,通过行为嵌入检索与评估策略,并基于离线Q值组合生成新策略。聚焦支持受限场景(无法进行值传播),发现复用效果关键取决于可用转移覆盖度。为平衡性能与计算成本,设计控制候选策略探索的组合策略。在确定性网格世界实验中,推理时组合可达到甚至超越从零训练的性能,同时显著提速。但当任务涉及长时序依赖需值传播时,性能下降,揭示离线复用的根本局限。
原文摘要 · Abstract (English)
Reinforcement learning (RL) policies are typically trained for fixed objectives, making reuse difficult when task requirements change. We study inference-time policy reuse: given a library of pre-trained policies and a new composite objective, can a high-quality policy be constructed entirely offline, without additional environment interaction? We introduce lever (Leveraging Efficient Vector Embeddings for Reusable policies), an end-to-end framework that retrieves relevant policies, evaluates them using behavioral embeddings, and composes new policies via offline Q-value composition. We focus on the support-limited regime, where no value propagation is possible, and show that the effectiveness of reuse depends critically on the coverage of available transitions. To balance performance and computational cost, lever proposes composition strategies that control the exploration of candidate policies. Experiments in deterministic GridWorld environments show that inference-time composition can match, and in some cases exceed, training-from-scratch performance while providing substantial speedups. At the same time, performance degrades when long-horizon dependencies require value propagation, highlighting a fundamental limitation of offline reuse.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。