通过分析残差流揭示安全微调导致误拒的原因，并提出新方法有效降低误拒率。
Residual Stream Analysis of Overfitting And Structural Disruptions
- 用基于PCA的FlowLens工具分析残差流几何，发现安全数据方差集中于少数成分。
- 安全数据占比从0%升至40%时，误拒率由63%增至84%，影响模型泛化能力。
- 提出方差集中损失（VCL）正则项，使误拒率下降超35个百分点，保持基准性能。
确保大语言模型既有益又无害面临重大挑战：在重复的安全数据集上微调时，不安全提示与标准拒绝模板配对，常导致对良性查询的误拒。我们首先量化该现象，发现安全数据的词元熵和二元组多样性（0.048）远低于通用指令数据。为揭示根本原因，引入稳定性的流透镜（FlowLens）——一种基于PCA的残差流几何分析工具，结果表明安全样本比例越高，方差越集中在少数成分，降低表征平滑性，引发误拒（安全数据占比从0%增至40%时，误拒率由63%升至84%）。基于此，提出方差集中损失（VCL），在中间层残差中惩罚过度方差集中。实验表明，VCL将误拒率降低超过35个百分点，同时在MMLU、GSM8K等通用基准上保持或提升性能。
原文摘要 · Abstract (English)
Ensuring that large language models (LLMs) remain both helpful and harmless poses a significant challenge: fine-tuning on repetitive safety datasets, where unsafe prompts are paired with standard refusal templates, often leads to false refusals, in which benign queries are declined. We first quantify this effect, showing that safety data exhibits substantially lower token entropy and 2-gram diversity (0.048) compared to general instruction data. To uncover the root cause, we introduce FlowLens, a stable PCA-based tool for residual-stream geometry analysis, and reveal that higher proportions of safety examples concentrate variance along a few components, reducing representational smoothness and driving false refusals (false refusal rate rises from 63 percent to 84 percent as safety data increases from 0 percent to 40 percent). Guided by these insights, we propose Variance Concentration Loss (VCL), an auxiliary regularizer that penalizes excessive variance concentration in mid-layer residuals. Empirical results demonstrate that VCL reduces false refusals by over 35 percentage points while maintaining or improving performance on general benchmarks such as MMLU and GSM8K.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。