arXiv:2607.03377cs.CLcs.AI2026-07KDD

用权重谱形状分析大模型,实现无数据、高效、可比的模型管理。

Spectral Signatures of Large Language Models

论文配图:Spectral Signatures of Large Language Models
图 1 · 摘自论文原文
  • 基于权重谱密度形状构建模型的紧凑谱签名
  • 能追踪模型演化、聚类相似模型、预测性能趋势
  • 无需数据、计算快、适合大规模模型分析

日益增长的公开大语言模型(LLMs)库给系统化管理与量化带来挑战,如模型溯源、许可合规和评估。传统任务基准不足以应对架构、规模和训练方式差异巨大的情况。本文基于重尾自正则化理论,提出一种基于谱形的度量方法,利用权重经验谱密度的形状作为模型的紧凑谱签名。该签名捕捉预训练模型的内在特性,且在微调后保持稳定,适用于模型级分析。该指标无需数据、计算高效、尺度不变,可实际用于大规模分析。我们构建了一个包含主流开源模型家族的大型多样化模型库,并系统性地在模型和下游任务上对比谱与非谱度量。结果表明,该谱签名可支持模型溯源、无监督聚类和性能量化。总体而言,该谱签名为大模型间的广泛性能趋势提供了有意义的代理,助力高效组织、比较与分析大规模模型集合。

原文摘要 · Abstract (English)

The rapidly growing repository of publicly available large language models (LLMs) presents significant challenges for systematic management and quantification at scale, such as model lineage tracing, licensing, and evaluation. However, task-specific benchmarks are insufficient for this setting, as LLMs differ widely in architectures, scales, and training procedures. To address this challenge, we adopt spectral shape-based metrics for managing and quantifying LLMs based on Heavy-Tailed Self-Regularization theory. Our approach uses the shape information of the weight empirical spectral density as a compact spectral signature of each model. This signature captures intrinsic properties of pretrained models and remains robust during post-training, making it suitable for model-level analysis. In addition, this metric is data-free, computationally-efficient, and scale-invariant, enabling large-scale analysis in practice. Moreover, we curate a large and diverse model corpus consisting of major open-source LLM families, and use it to systematically benchmark spectral and non-spectral metrics across models and downstream tasks. We show that our spectral signature supports the tracking of the model lineage, the unsupervised clustering of similar models, and the quantification of the model performance. Overall, the proposed spectral signature provides a meaningful proxy for broad performance trends across LLMs, enabling efficient organization, comparison, and analysis of large model collections.

大模型分析谱签名模型溯源无数据评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。