用谱饱和度指标判断少样本标注何时不再有效。
The Geometry of Saturation: Effective Rank Predicts When Labels Stop Helping in Few-Shot Classification
- 提出谱饱和指数 $S(K)$，衡量每标签的探索效率。
- 当 $S(K)<0.02$ 时，准确率提升趋近于零，验证集上分类准确率达 78.7%。
- 适用于冻结主干网络的少样本学习，计算仅需约 1 毫秒。
少样本标签获取缺乏标签无关的停止信号：现有停止标准或需保留验证集（违背少样本前提），或依赖无理论依据的启发式方法。本文引入谱饱和指数 $S(K) = \mathrm{erank}(\hatΣ_W^{(K)})/K$，即归一化类内协方差的指数谱熵，衡量每标签的探索速率。当 $S(K)$ 低于固定阈值 $τ=0.02$ 时，谱子空间趋于饱和，边际增益消失。在 49 个真实任务（二分类、5 类、10 类）和三种冻结主干网络（PCA-50、CLIP ViT-B/32、DINOv2 ViT-S/14）上，$S(K)$ 与支持集翻倍带来的边际增益高度相关（$ρ_{\text{pool}}=0.6366$, $p=2.9×10^{-57}$，置信区间 [0.551,0.720]）；固定 $τ=0.02$ 的决策分类器在集群自举下达到 $\mathrm{AUC}=0.787$（95% CI [0.713,0.860]），对 $ΔA>1\%$ 的有效增益具有高召回率。偏相关分析控制 $\log K$ 后仍得 $ρ_{\text{partial}}=0.324$（$p=1.65×10^{-13}$），证明 $S(K)$ 包含超越 $K$ 共同依赖的谱信息。理论表明 $K_{\text{sat}}≈\mathrm{erank}(Σ_W)/τ$，且 $τ=0.02$ 位于 Nakkiran 等（2021）所述第一、第二下降区间的边界，小 $K$ 时的 $S(K)$ 小幅上翘由样本有效秩的 $O(1/K)$ 偏差解释。对于无正则化的线性探测器（$C=∞$），建议在 $S(K)<0.02$ 时终止（PCA-50，硬停止），或监控 $S(K)$ 从 ~0.3 下降至 0.05（基础模型，收益递减信号），计算成本约为 $d=50$ 时 1 毫秒。
原文摘要 · Abstract (English)
Few-shot label acquisition lacks a label-free signal for when additional labels cease to improve accuracy: existing stopping criteria either require a held-out validation set (violating the few-shot premise) or rely on theoretically ungrounded heuristics, so we introduce the spectral saturation index $S(K)=\mathrm{erank}(\hatΣ_W^{(K)})/K$, the exponential spectral entropy of the pooled within-class covariance normalized by per-class support size $K$, which measures the exploration rate per label and falls below a fixed threshold $τ=0.02$ once the explored spectral subspace saturates and marginal accuracy gains vanish; across 49 real tasks (binary, 5-way, 10-way) and three frozen backbones (PCA-50, CLIP ViT-B/32, DINOv2 ViT-S/14), $S(K)$ correlates strongly with the marginal gain on doubling the support set ($ρ_{\text{pool}}=0.6366$, $p=2.9\times10^{-57}$, cluster-bootstrap 95\% CI $[0.551,0.720]$), a fixed $τ=0.02$ classifies stop/continue decisions with cluster-bootstrap $\mathrm{AUC}=0.787$ (95\% CI $[0.713,0.860]$) with high recall on meaningful gains ($ΔA>1\%$), and a partial correlation controlling for $\log K$ yields $ρ_{\text{partial}}=0.324$ ($p=1.65\times10^{-13}$), confirming $S(K)$ carries spectral information beyond shared $K$-dependence; theory predicts this from first principles, since the population effective rank sets the saturation scale $K_{\text{sat}}\approx\mathrm{erank}(Σ_W)/τ$, $τ=0.02$ sits at the boundary between the first and second descent (Nakkiran et al., 2021), and $O(1/K)$ bias in the sample effective rank explains the small-$K$ hump in $S(K)$; for unregularized linear probes ($C=\infty$), practitioners should halt when $S(K)<0.02$ (PCA-50, hard stop) or monitor $S(K)$ dropping from $\sim0.3\to0.05$ (foundation models, diminishing-returns signal), with computation costing $\sim1$ ms at $d=50$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。