让大模型像变色龙一样随输入切换行为,无需重训练
Revealing Behavioral Plasticity in Large Language Models: A Token-Conditional Perspective
- 用特定前缀触发模型在推理时动态切换行为模式
- 通过强化学习使行为切换稳定,不降低原有能力
- 适合需要灵活应对多任务的场景,如问答与推理混合
本文揭示大型语言模型具备内在的行为可塑性——如同变色龙随环境改变体色——可通过条件化生成和强化学习加以暴露与固化。具体而言,通过选取展现期望行为的响应中采样的特定前缀进行条件生成,模型可在推理阶段无缝切换行为模式(如从逐步推理转为直接作答),而无需重新训练。基于此,我们提出一种名为 ToCoRL(Token-Conditioned Reinforcement Learning)的系统性框架,利用强化学习将这种类变色龙的可塑性内化为稳定的、可学习的行为模式。ToCoRL 以条件化生成引导探索并持续优化利用,促使合适行为涌现。大量实验表明,ToCoRL 实现了精准的行为控制且未导致性能下降。值得注意的是,原本在复杂数学任务上表现优异的推理型大模型,可被有效适配以擅长事实性问答,而此前其逐步推理模式曾制约该能力。
原文摘要 · Abstract (English)
In this work, we reveal that Large Language Models (LLMs) possess intrinsic behavioral plasticity-akin to chameleons adapting their coloration to environmental cues-that can be exposed through token-conditional generation and stabilized via reinforcement learning. Specifically, by conditioning generation on carefully selected token prefixes sampled from responses exhibiting desired behaviors, LLMs seamlessly adapt their behavioral modes at inference time (e.g., switching from step-by-step reasoning to direct answering) without retraining. Based on this insight, we propose Token-Conditioned Reinforcement Learning (ToCoRL), a principled framework that leverages RL to internalize this chameleon-like plasticity, transforming transient inference-time adaptations into stable and learnable behavioral patterns. ToCoRL guides exploration with token-conditional generation and keep enhancing exploitation, enabling emergence of appropriate behaviors. Extensive experiments show that ToCoRL enables precise behavioral control without capability degradation. Notably, we show that large reasoning models, while performing strongly on complex mathematics, can be effectively adapted to excel at factual question answering, which was a capability previously hindered by their step-by-step reasoning patterns.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。