用简单线性生成器提升模型权重空间探测效率与效果
Deep Linear Probe Generators for Weight Space Learning
- 引入共享线性生成器，让探测输入更结构化
- 比现有最佳方法快30到1000倍，且性能更优
- 适合需要高效分析模型权重的研究者
权重空间学习旨在从神经网络权重中提取训练数据或泛化误差等信息。现有方法直接从高维权重中学习，但面临维度高和神经元置换对称性等问题。另一种思路是探针（Probing）：用一组可学习的输入（探针）通过模型，再训练预测器以对应输出。尽管探针通常不独立使用，但初步实验发现基础探针基线表现意外良好。然而我们发现当前探针学习策略效果不佳。为此提出深线性探针生成器（ProbeGen），在探针学习中引入共享的深层线性生成模块，提供结构化先验，减少过拟合。虽然结构简单，但性能显著优于当前最优方法，且计算效率极高，所需浮点运算量仅为其他顶尖方法的1/30至1/1000。
原文摘要 · Abstract (English)
Weight space learning aims to extract information about a neural network, such as its training dataset or generalization error. Recent approaches learn directly from model weights, but this presents many challenges as weights are high-dimensional and include permutation symmetries between neurons. An alternative approach, Probing, represents a model by passing a set of learned inputs (probes) through the model, and training a predictor on top of the corresponding outputs. Although probing is typically not used as a stand alone approach, our preliminary experiment found that a vanilla probing baseline worked surprisingly well. However, we discover that current probe learning strategies are ineffective. We therefore propose Deep Linear Probe Generators (ProbeGen), a simple and effective modification to probing approaches. ProbeGen adds a shared generator module with a deep linear architecture, providing an inductive bias towards structured probes thus reducing overfitting. While simple, ProbeGen performs significantly better than the state-of-the-art and is very efficient, requiring between 30 to 1000 times fewer FLOPs than other top approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。