arXiv:2606.27242cs.LGcs.CL2026-06中稿 · ICML被引 1

无需训练即可选大模型源数据,靠更新方向几何判断迁移效果。

The Geometry of Updates: Fisher Alignment at Vocabulary Scale

论文配图:The Geometry of Updates: Fisher Alignment at Vocabulary Scale
图 1 · 摘自论文原文
  • 用头参数更新的Fisher相似度替代表示相似性,避免无效判断。
  • 单次流式计算实现128~256维头更新对齐,仅需16KB签名。
  • 适合研究任务迁移本质,尤其激活不相似但更新方向相关的场景。

在共享词表的大语言模型家族中(如SMILES、蛋白质和基因组序列),候选语料库共用分词器但目标预测不同,导致无监督选择源数据困难。传统表示相似性指标在缺乏标签条件误差结构假设时失效,而经典更新几何度量在词汇规模下计算成本过高。本文发现,在共享输出头设置下,表示相似性(如CKA)无法识别迁移潜力:模型可拥有相同表示却有正交的头更新。关键突破在于,头的Fisher对齐等于联合激活-误差空间中核均值嵌入间的余弦值,揭示激活、误差及耦合因素,无需显式构造费舍尔矩阵。FisherSketch直接在单次流式遍历中估计该余弦,使K=128,256的头更新对齐成为可能,仅需16KB任务签名(m=4096)和每任务192KB流式状态,小到可与模型哈希并存,却编码了迁移相关更新结构。除源选择外,同一签名与边缘分布还可诊断任务相似性由激活、误差或其耦合驱动;通过共享参数和内部层验证,以及基于Llama-3.1-8B的表述符迁移实验,证明当激活相似性无法区分任务时,FisherSketch仍具信息量。

原文摘要 · Abstract (English)

Training-free source selection for LLM families with shared vocabularies arises in scientific string domains such as SMILES, protein, and genomic sequences, where candidate corpora share a tokenizer but differ in prediction targets. This creates an activation-dark regime: representation-similarity metrics can be uninformative without assumptions about label-conditioned error geometry, while classical update-geometry metrics are computationally prohibitive at vocabulary scale. We show that, in a shared-output head setting, representation metrics (e.g., CKA) are non-identifiable for transfer; models can share identical representations yet have orthogonal head updates. The key identity is that head Fisher alignment is exactly a cosine between kernel mean embeddings in the joint activation-error space, exposing activation, error, and coupling factors rather than requiring a materialized Fisher matrix. FisherSketch estimates this cosine directly in a single streaming pass, making K=128,256 head Fisher alignment practical with a 16 KB task signature (m=4096) and a 192 KB per-task streaming state, small enough to store next to a model hash, but encoding transfer-relevant update structure. Beyond source selection, the same signatures and marginals provide a diagnostic instrument for studying whether LLM task similarity is driven by activations, errors, or their coupling; shared-parameter and internal-layer validations, together with Llama-3.1-8B verbalizer-shift experiments, show that FisherSketch remains informative when activation similarity cannot distinguish tasks.

大模型迁移Fisher几何无训练选择任务诊断

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。