arXiv:2503.12923cs.LGcs.SI2025-03被引 1

用大模型生成动态权重,让AI持续学习不遗忘。

Lifelong Reinforcement Learning with Similarity-Driven Weighting by Large Models

  • 用大模型提前生成任务相似度和权重函数,动态调节学习策略。
  • 在Atari和MiniHack上表现超越现有方法,有效缓解遗忘问题。
  • 适合需要长期学习与知识迁移的智能体场景。

终身强化学习(LRL)在处理序列任务方面潜力巨大,但仍面临灾难性遗忘和知识迁移困难等挑战。为此,我们提出一种新框架SDW(相似性驱动加权框架),利用大语言模型生成的动态函数精准控制训练过程。其核心是两个由大模型预先生成的函数:任务相似度函数用于从任务描述中提取状态、动作、奖励等多维特征,量化任务间的相似性;权重计算函数则基于相似度信息动态生成关键训练参数,包括重放缓冲区中旧任务数据的比例和损失函数中的策略一致性权重,实现新旧任务学习间的自适应平衡。通过在训练前离线生成函数代码,而非依赖训练时的大模型推理,SDW显著降低计算开销,同时保持高效性。在Atari和MiniHack序列任务上的实验表明,SDW显著优于现有终身强化学习方法。

原文摘要 · Abstract (English)

Lifelong Reinforcement Learning (LRL) holds significant potential for addressing sequential tasks, but it still faces considerable challenges. A key difficulty lies in effectively preventing catastrophic forgetting and facilitating knowledge transfer while maintaining reliable decision-making performance across subsequent tasks in dynamic environments. To tackle this, we propose a novel framework, SDW (Similarity-Driven Weighting Framework), which leverages large-language-model-generated dynamic functions to precisely control the training process. The core of SDW lies in two functions pre-generated by large models: the task similarity function and the weight computation function. The task similarity function extracts multidimensional features from task descriptions to quantify the similarities and differences between tasks in terms of states, actions, and rewards. The weight computation function dynamically generates critical training parameters based on the similarity information, including the proportion of old task data stored in the Replay Buffer and the strategy consistency weight in the loss function, enabling an adaptive balance between learning new tasks and transferring knowledge from previous tasks. By generating function code offline prior to training, rather than relying on large-model inference during the training process, the SDW framework reduces computational overhead while maintaining efficiency in sequential task scenarios. Experimental results on Atari and MiniHack sequential tasks demonstrate that SDW significantly outperforms existing lifelong reinforcement learning methods.

终身学习强化学习大模型应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。