发现大激活未必有害，提出新方法平衡模型性能与激活抑制
A Refined Analysis of Massive Activations in LLMs
- 分析多种LLM架构下的大激活现象，突破传统认知
- 验证抑制大激活不必然导致困惑度飙升或任务性能下降
- 提出TVR+KV偏置或DyT的混合策略，兼顾性能与稳定性
为应对低精度训练与量化中的挑战，大规模激活在大语言模型（LLMs）中受到关注。然而现有研究范围有限，跨架构通用性不明确。本文对多种架构的LLM进行了广泛分析，涵盖GLU与非GLU结构。结果挑战了多项既有假设：(1) 并非所有大激活都有害，抑制它们不会引发困惑度剧增或下游任务性能崩溃；(2) 现有缓解策略如注意力键值偏置具有模型依赖性，在某些情况下无效。因此我们探索新型混合缓解策略，特别是将目标方差重缩放（TVR）与注意力键值偏置或动态双曲正切（DyT）结合，在所考察场景中成功平衡了大激活抑制与下游性能保持。代码已开源：https://github.com/bluorion-com/refine_massive_activations。
原文摘要 · Abstract (English)
Motivated in part by their relevance for low-precision training and quantization, massive activations in large language models (LLMs) have recently emerged as a topic of interest. However, existing analyses are limited in scope, and generalizability across architectures is unclear. This paper helps address some of these gaps by conducting an analysis of massive activations across a broad range of LLMs, including both GLU-based and non-GLU-based architectures. Our findings challenge several prior assumptions, most importantly: (1) not all massive activations are detrimental, i.e. suppressing them does not lead to an explosion of perplexity or a collapse in downstream task performance; (2) proposed mitigation strategies such as Attention KV bias are model-specific and ineffective in certain cases. We consequently investigate novel hybrid mitigation strategies; in particular pairing Target Variance Rescaling (TVR) with Attention KV bias or Dynamic Tanh (DyT) successfully balances the mitigation of massive activations with preserved downstream model performance in the scenarios we investigated. Our code is available at: https://github.com/bluorion-com/refine_massive_activations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。