让大模型自己生成训练数据并更新权重，实现持续自我适应。
Self-Adapting Language Models
- 模型自动生成微调数据和更新指令，直接修改自身参数。
- 通过强化学习优化自编辑效果，下游任务性能提升显著。
- 无需额外模块，适合需要持续学习的场景。
大语言模型（LLMs）虽强大但静态，缺乏对新任务或知识的自适应能力。我们提出自适应语言模型（SEAL），使模型能根据新输入生成自编辑内容——包括信息重构、超参数设定或调用工具进行数据增强与梯度更新。经监督微调（SFT），这些自编辑带来持久的权重更新。为训练模型生成有效自编辑，我们采用以更新后模型的下游性能为奖励信号的强化学习循环。相较于依赖独立适配模块或辅助网络的方法，SEAL直接利用模型自身生成来控制适应过程。在知识融入与少样本泛化实验中，SEAL展现出实现自主适应语言模型的潜力。代码与网页详见 https://jyopari.github.io/posts/seal。
原文摘要 · Abstract (English)
Large language models (LLMs) are powerful but static; they lack mechanisms to adapt their weights in response to new tasks, knowledge, or examples. We introduce Self-Adapting LLMs (SEAL), a framework that enables LLMs to self-adapt by generating their own finetuning data and update directives. Given a new input, the model produces a self-edit-a generation that may restructure the information in different ways, specify optimization hyperparameters, or invoke tools for data augmentation and gradient-based updates. Through supervised finetuning (SFT), these self-edits result in persistent weight updates, enabling lasting adaptation. To train the model to produce effective self-edits, we use a reinforcement learning loop with the downstream performance of the updated model as the reward signal. Unlike prior approaches that rely on separate adaptation modules or auxiliary networks, SEAL directly uses the model's own generation to control its adaptation process. Experiments on knowledge incorporation and few-shot generalization show that SEAL is a promising step toward language models capable of self-directed adaptation. Our website and code is available at https://jyopari.github.io/posts/seal.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。