用缓存优化大模型引导强化学习，省下大量计算却几乎不丢性能。
Cache-Efficient Posterior Sampling for Reinforcement Learning with LLM-Derived Priors Across Discrete and Continuous Domains
- 自适应缓存机制通过策略表现反向优化缓存参数。
- 减少3.8到4.7倍大模型调用，延迟降低4到12倍。
- 适合算力有限环境下做文本与连续控制任务的强化学习。
将大语言模型（LLMs）作为先验知识引入强化学习（RL）虽具优势，但计算开销巨大。本文提出一种原则性的缓存高效后验采样框架，显著降低计算成本同时保持高性能。核心是自适应缓存机制，缓存参数通过代理梯度从策略表现中元优化。该方法在离散文本环境（如TextWorld、ALFWorld）和连续控制域（如MuJoCo）均有效，实现3.8–4.7倍的LLM查询减少，中位延迟降低4.0–12.0倍（消费级GPU上85–93毫秒），性能保留96–98%。理论分析提供近似质量的KL散度边界，并经实证验证。框架扩展至离线强化学习，其CQL-Prior变体提升性能14–29%，训练时间缩短38–40%。八项任务的全面评估证明了该方法在资源受限场景下的通用性与实用性。
原文摘要 · Abstract (English)
Integrating large language models (LLMs) as priors in reinforcement learning (RL) offers significant advantages but comes with substantial computational costs. We present a principled cache-efficient framework for posterior sampling with LLM-derived priors that dramatically reduces these costs while maintaining high performance. At the core of our approach is an adaptive caching mechanism, where cache parameters are meta-optimized using surrogate gradients derived from policy performance. This design enables efficient inference across both discrete text environments (e.g., TextWorld, ALFWorld) and continuous control domains (e.g., MuJoCo), achieving a 3.8--4.7$\times$ reduction in LLM queries and 4.0--12.0$\times$ lower median latencies (85--93\,ms on a consumer GPU) while retaining 96--98\% of uncached performance. Our theoretical analysis provides KL divergence bounds on approximation quality, validated empirically. The framework extends to offline RL, where our CQL-Prior variant improves performance by 14--29\% and reduces training time by 38--40\%. Extensive evaluations across a diverse suite of eight tasks demonstrate the generalizability and practical viability of LLM-guided RL in resource-constrained settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。