arXiv:2605.22005cs.LGcs.AI2026-05

仅用五行代码分析模型权重,就能发现大模型学到了什么,包括不该学的内容。

Check Your LLM's Secret Dictionary! Five Lines of Code Reveal What Your LLM Learned (Including What It Shouldn't Have)

  • 通过分解模型输出层权重矩阵,提取可解释的语义子空间。
  • 发现GPT、Gemma、Qwen等模型在词汇分布上存在系统性差异,揭示训练数据特征。
  • 可无须运行模型就检测到有害词汇和漏洞词,适合安全审计与模型优化。

我们证明,对基于Transformer的大语言模型的lm_head权重矩阵进行奇异值分解——仅需五行PyTorch代码且无需模型推理——即可直接从模型权重中揭示可解释的语义子空间。每个左奇异向量对应一组最易被选中的词汇项,其聚类结构暴露了模型的训练数据构成与数据筛选理念。分析GPT-OSS-120B、Gemma-2-2B和Qwen2.5-1.5B发现,不同模型的奇异值谱与词汇聚类结构具有系统性差异:GPT呈现功能分层的子空间结构;Gemma以19世纪前英语拼写为主,形成阶梯式聚类,可能提升输出可控性;Qwen则表现出广泛多语言覆盖,同时存在作者认定为伦理不恰当的词汇子空间。对比base与instruct版本表明,此类问题源于预训练阶段,后训练对齐无法消除。我们提出词汇聚类得分(VCS)量化子空间一致性,引入加权投影得分(WPS)作为静态漏洞词检测器;应用WPS于GPT-OSS-120B成功复现了广为人知的汉字表漏洞词shokubutsu-hyakka-tsu(ID 137606),无需任何模型推理。本文提出问题词汇成因分类法,呼吁将lm_head SVD分析纳入发布前安全审计流程,并提示其在分词器优化与更可控模型设计中的潜力。

原文摘要 · Abstract (English)

We show that singular value decomposition of the lm_head} weight matrix of a transformer-based large language model -- requiring only five lines of PyTorch and no model inference -- reveals interpretable semantic subspaces directly from the model weights. Each left singular vector identifies the vocabulary tokens most readily selected when the hidden state aligns with the corresponding singular direction; inspecting these clusters exposes the model's training data composition and curation philosophy. Analysing GPT-OSS-120B, Gemma-2-2B, and Qwen2.5-1.5B, we find that singular value spectra and vocabulary cluster structures differ systematically across models: GPT exhibits a graduated hierarchy of functionally differentiated subspaces; Gemma is dominated by pre-nineteenth-century English orthography, forming a stepwise clustering structure that may contribute to high output controllability; and Qwen exhibits broad multilingual coverage alongside subspaces whose vocabulary the authors have determined to be ethically inappropriate for direct publication. Base-instruct comparison reveals that ethically concerning subspaces originate in pretraining and are not removed by post-training alignment. We introduce the Vocabulary Cluster Score (VCS) to quantify subspace coherence, and the Weighted Projection Score (WPS) as a static glitch token detector; applying WPS to GPT-OSS-120B recovers shokubutsu-hyakka-tsu (ID 137606), a well-known glitch token widely reported in the CJK language community, without any model inference. We propose a taxonomy of root causes for problematic vocabulary content and call for lm_head} SVD analysis to be adopted as a standard pre-release safety auditing step. Our findings further suggest directions toward SVD-guided tokenizer optimisation and more controllable LLM design.

模型审计安全分析可解释性权重分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。