arXiv:2607.13395cs.LG2026-07

不更新权重,通过调整连接方式实现大模型突然能力提升。

Self-Improving is Often Sudden: Enlightenment-style Finetuning for Large-Scale Models

论文配图:Self-Improving is Often Sudden: Enlightenment-style Finetuning for Large-Scale Models
图 1 · 摘自论文原文
  • 用新连接方式替代传统注意力调整,无须训练即可激活模型潜能。
  • 在多个模型和任务上显著提升性能,语言与视觉任务均有效。
  • 适合希望快速优化大模型但无法重训的研究者或工程师。

大规模基础模型的自主自进化研究日益受到关注。受人类大脑中‘顿悟’或‘恍然大悟’现象启发,我们假设大模型也存在类似的顿悟现象——即潜在的突发能力跃升。为此,提出 Enlightenment,一种无需训练的后调优范式。该方法通过修改关键模块/层的连接路径(而非权重),突破现有训练自由方法主要依赖注意力权重调整的局限。针对大语言模型,提出注意力头混接捷径,将初始注意力头输出与所有目标头相连,并采用自适应缩放因子初始化策略重新校准注意力;针对视觉-语言模型,在解码器层的残差连接上引入轻量级标量调制因子,调控信息流。大量实验表明,Enlightenment 能高效释放预训练网络的潜在能力,在多种基准测试与模型上均取得显著性能提升。

原文摘要 · Abstract (English)

The pursuit of autonomously self-improving models has attracted growing interest in the era of large-scale foundation models. Drawing inspiration from the concept of "enlightenment" or "aha moment" in human brain, we hypothesize that large models exhibit an analogous enlightenment phenomenon-a latent capacity for sudden capability boost. Then, we propose Enlightenment, a novel training-free post-tuning paradigm for large-scale models. Our approach modifies shortcuts for key modules/layers without weight updates, while existing training-free ones predominantly manipulate attention weights. We introduce two architecture-specific instantiations: i) For large language models, we propose attention head-mixing shortcuts that recalibrate attention weights by linking the initial attention head's output to all other target heads, modulated by an adaptive scaling factor initialization strategy. ii) For vision-language models, we apply a lightweight scalar-modulated factor to residual connections in the decoder layers, regulating information flow. Extensive experiments show that Enlightenment efficiently unlocks the latent potential of pre-trained networks, yielding remarkable performance improvements across diverse benchmarks and models.

大模型优化无训练调优顿悟机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。