arXiv:2605.11608cs.CLcs.AI2026-05

PRISM通过几何分解揭示大模型微调后的风险来源,指导修复方向。

PRISM: A Geometric Risk Bound that Decomposes Drift into Scale, Shape, and Head

论文配图:PRISM: A Geometric Risk Bound that Decomposes Drift into Scale, Shape, and Head
图 1 · 摘自论文原文
  • 基于输出头与骨干结构的几何特性,构建风险上界。
  • 将漂移分解为尺度、形状、头部三维度,对应不同失效模式。
  • 可定位问题根源并指导修复,适用于量化与适配优化场景。

对比训练后的大语言模型变体(如量化、LoRA适配、蒸馏模型)需要一种诊断工具,以识别其表示漂移的类型,而不仅是是否退化。现有相似性度量如CKA和SVCCA可检测退化,但无法直接关联漂移与风险或机制。我们提出PRISM(Proxy Risk Inference via Structural Mapping),利用大模型线性输出头及骨干网络近等距结构,推导出目标模型与训练后变体之间交叉熵风险差的闭式上界。该上界可用于变体排序,并将漂移分解为三个独立可测维度:尺度失配、形状失配和头部偏离。每个维度对应不同失效模式,包括低比特量化下的形状畸变、LoRA遗忘导致的尺度分离、以及GGUF k-量化下的头部偏离。因此,主导轴可指引修复方向,而非仅发出退化警告。由于形状项可微分,该几何结构亦可用作训练时正则化项,抑制灾难性遗忘。在两个模型族和五个基准上,PRISM对量化变体的排序平均斯皮尔曼相关系数达0.820,对LoRA遗忘的排序相关系数为0.831;其轴引导的形状正则化器在缓解下游遗忘方面整体优于经验回放。

原文摘要 · Abstract (English)

Comparing post-training LLM variants, such as quantized, LoRA-adapted, and distilled models, requires a diagnostic that identifies how a variant has drifted, not only whether it has degraded. Existing similarity scores such as CKA and SVCCA can flag degradation, but they do not directly link representation drift to risk or mechanism. We propose PRISM, Proxy Risk Inference via Structural Mapping, which exploits the linear output head of LLMs and the empirically near-isometric structure of their backbones to derive a closed-form upper bound on the cross-entropy risk gap between a target model and a post-training variant. The bound is calibrated for variant ranking and decomposes drift into three independently measurable axes: scale mismatch, shape mismatch, and head divergence. Each axis corresponds to a distinct failure mode, including shape distortion under low-bit quantization, scale separability under LoRA forgetting, and head divergence under GGUF k-quantization. As a result, the dominant axis suggests a remediation direction rather than merely raising a degradation flag. Because the shape term is differentiable, the same geometry can also serve as a training-time regularizer against catastrophic forgetting. Across two model families and five benchmarks, PRISM ranks variants with mean Spearman correlations of 0.820 for post-training quantization and 0.831 for LoRA forgetting, and its axis-guided shape regularizer outperforms experience replay in aggregate at mitigating downstream forgetting.

大模型风险评估量化几何分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。