arXiv:2604.20556cs.CLcs.AI2026-04

提出可适配任意大模型架构的层级分析框架,定位任务执行起点与脆弱层。

LayerTracer: A Joint Task-Particle and Vulnerable-Layer Analysis framework for Arbitrary Large Language Model Architectures

  • 通过逐层提取隐藏状态并映射概率分布,联合分析任务粒子与层脆弱性。
  • 任务粒子多位于深层,大参数模型具有更强层级鲁棒性。
  • 适用于混合架构设计优化,提升模型可解释性与结构设计科学性。

当前大语言模型(LLMs)架构多样,包括传统Transformer、GateDeltaNet和Mamba等。然而,不同架构中层次表征演化规律、任务知识形成位置及网络鲁棒性瓶颈机制仍不清晰,制约了混合架构设计与模型优化。本文提出LayerTracer,一个与架构无关的端到端分析框架,兼容任意LLM架构。通过逐层提取隐藏状态并映射至词汇概率分布,实现任务粒子定位与层脆弱性量化联合分析。任务粒子定义为目标词概率首次显著上升的关键层,代表任务执行起始点;脆弱层则为在掩码扰动前后输出分布间Jensen-Shannon(JS)散度最大的层,反映其对扰动的敏感性。在不同参数规模模型上的实验表明,任务粒子主要出现在深层,而大参数模型展现出更强的层级鲁棒性。LayerTracer为层划分、模块比例与门控切换提供科学依据,有效优化模型性能,精准定位任务有效层与稳定性瓶颈,支持大模型结构设计与可解释性研究。

原文摘要 · Abstract (English)

Currently, Large Language Models (LLMs) feature a diversified architectural landscape, including traditional Transformer, GateDeltaNet, and Mamba. However, the evolutionary laws of hierarchical representations, task knowledge formation positions, and network robustness bottleneck mechanisms in various LLM architectures remain unclear, posing core challenges for hybrid architecture design and model optimization. This paper proposes LayerTracer, an architecture-agnostic end-to-end analysis framework compatible with any LLM architecture. By extracting hidden states layer-by-layer and mapping them to vocabulary probability distributions, it achieves joint analysis of task particle localization and layer vulnerability quantification. We define the task particle as the key layer where the target token probability first rises significantly, representing the model's task execution starting point, and the vulnerable layer is defined as the layer with the maximum Jensen-Shannon (JS) divergence between output distributions before and after mask perturbation, reflecting its sensitivity to disturbances. Experiments on models of different parameter scales show that task particles mainly appear in the deep layers of the model regardless of parameter size, while larger-parameter models exhibit stronger hierarchical robustness. LayerTracer provides a scientific basis for layer division, module ratio, and gating switching of hybrid architectures, effectively optimizing model performance. It accurately locates task-effective layers and stability bottlenecks, offering universal support for LLM structure design and interpretability research.

大模型分析可解释性层脆弱性架构设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。