让语言模型学会按需更新,提升适应效率并减少无效参数修改。
PARALLEL: A Prefrontal-Aligned Reinforcement inspired Approach for Language-Model Learning under Explicit Limits

- 基于前额叶控制机制设计双信号控制器,动态调节每条样本的更新强度。
- 在多个任务上保持94.1%~99.2%全量更新性能,生成任务保留96.9%~98.6%的ROUGE分数。
- 适合需要持续更新但资源受限的部署场景,如边缘设备或实时推理系统。
当前语言模型在多种任务中表现优异,但传统微调对所有样本统一更新,忽视局部收益差异。本文提出PARALLEL,一种受前额叶调控启发的强化学习方法,将目标相关与不确定性信息作为独立控制信号,结合当前模型表示,通过即时效用-成本反馈决定样本级更新强度。该方法可学习何时以及多强地对每个样本进行调整,优先有益更新并限制冗余参数变化。相比选择性基线,PARALLEL更高效利用可用更新,在多项任务上保持94.1%~99.2%的全量微调性能。在XSum和CNN/DailyMail上,其生成指标保留96.9%~98.6%的ROUGE-1/2与98.8%~98.9%的ROUGE-L分数。在相同累计适应时间或GPU能耗下,其在ARC任务上取得更高准确率,并展现出更稳定的后期适应轨迹。结果表明,学习样本级更新时机与强度,能实现高效且稳定的后部署流式适应。
原文摘要 · Abstract (English)
Recent language models achieve strong performance across a variety of tasks, but conventional adaptation applies updates uniformly across training samples regardless of their local update benefit. We propose PARALLEL, a prefrontal-aligned reinforcement inspired approach for language-model learning. Inspired by the complementary roles of goal-related and uncertainty-related control, PARALLEL represents these forms of information as separate controller signals and combines them with the current model representation. A reinforcement-inspired controller assigns sample-dependent update intensity using immediate utility-cost feedback. PARALLEL therefore learns when and how strongly to adapt to each sample, prioritizing beneficial updates while limiting unnecessary parameter changes. PARALLEL uses available updates more efficiently than selective baselines while retaining 94.1--99.2\% of Full-adaptation performance. Beyond multiple-choice reasoning, experiments on XSum and CNN/DailyMail show that PARALLEL retains 96.9--98.6\% of the ROUGE-1 and ROUGE-2 scores achieved by Full adaptation and 98.8--98.9\% of the corresponding ROUGE-L scores. When compared at the same cumulative adaptation time or GPU energy, PARALLEL achieves higher ARC accuracy and exhibits a more stable late-stage adaptation trajectory than Full adaptation in the representative run. These results show that learning when and how strongly to update each sample supports stable and efficient post-deployment stream adaptation while avoiding unnecessary updates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。