发现模型权重与激活的线性结构是局部动态的，非全局固定。
Recoverable but Not Stationary:Local Linear Structures in Weights and Activations
- 通过梯度轨迹发现局部低秩任务方向，可捕捉77%的参数恢复位移。
- 固定基底无法捕获恢复方向，而初始更新轨迹能有效表示变化中的几何结构。
- 适用于研究大模型微调、参数高效训练及激活操控的科研人员。
任务向量、LoRA、激活操控和预训练权重附近的随机搜索均表明，学习行为可通过线性方向控制。我们探究这些线性结构的真实存在性及其作用尺度。在合成多任务Transformer及在DistilGPT-2/GPT-2上的LoRA适配器中，发现强局部低秩任务梯度结构，但拒绝固定任务平面假设：静态基底无法捕捉恢复方向，有效基底在100步内显著漂移。然而，前几次恢复更新构成的轨迹前缀基底可捕获77%的LoRA恢复位移。我们提出随机搜索理论，基于高斯局部线性定理，解释了高维空间中随机参数搜索的有效性。还研究了参数扰动与激活操控的关系：单步梯度更新产生的激活偏移与标注对比的CAA操控向量夹角余弦为0.58，在Qwen-0.5B BoolQ任务上产生类似操控效果。实验验证于合成Transformer与大语言模型。结果表明，训练网络中的线性结构并非全局任务方向，而是部分跨参数与激活空间持续存在的动态局部几何。
原文摘要 · Abstract (English)
Task vectors, LoRA, activation steering, and random search around pretrained weights all suggest that learned behaviour can be controlled by linear directions. We ask which linear structures actually exist and on what scale. In a synthetic multitask transformer and LoRA adapters on DistilGPT-2 / GPT-2 we find strong local low-rank task-gradient structure but reject the fixed-task-plane hypothesis: static bases miss the recovery direction, and the useful basis drifts substantially within 100 steps. However, the first recovery updates form a trajectory-prefix basis capturing 77% of the LoRA recovery displacement. We develop random search theory with a Gaussian local-linear theorem that justifies the effectiveness of random parameter search even in very high dimensions. We also study the relation between parameter perturbations and activation steering: a single gradient step produces an activation shift with 0.58 cosine to a labelled-contrast CAA steering vector, with a similar steering effect on Qwen-0.5B BoolQ statements. We validate our results with experiments on synthetic Transformers and LLMs. Our results suggest that linear structures in trained networks are not global task directions, but evolving local geometries that partially persist across parameter and activation spaces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。