发现预训练提示在迁移后可能失效,需在适配前后评估语义价值。
When Semantics Saturate or Emerge: Adaptation-Conditional Semantic Utility in Source-Free Cross-Domain Few-Shot Learning
- 对比适配前后的详细描述效果,发现语义效用会饱和或新生
- 在欧空局和作物病害数据集上,细节描述增益从21.54%降至2.96%
- 适用于少样本跨域迁移学习中的提示工程优化与模型评估
在无源跨域少样本学习中,语言描述通常依据冻结视觉-语言模型的零样本准确率选择。本文通过严格配对实验,比较了通用类别名模板与固定详细类别描述在视觉低秩适配(LoRA)前后的表现。在EuroSAT、CropDisease、ISIC和ChestX数据集上,定义Δ_zero与Δ_loRA分别为适配前后详细描述减基线的准确率差值。结果呈现两种典型模式:在语义饱和情形下,Δ_zero > 0但0 < Δ_loRA ≪ Δ_zero,如在EuroSAT和CropDisease上,初始增益8.13–21.54个百分点收缩至0.69–2.96个百分点;在语义涌现情形下,Δ_zero ≤ 0但Δ_loRA > 0,如在ISIC和ChestX上,详细描述仅在视觉表示更新后才变得更有用。训练轨迹与样本级分解显示,饱和由基线LoRA恢复原有细节语义所驱动,而涌现则关联于预测转变与新出现的仅靠细节正确判断。多重控制实验(无固定点、另一CLIP主干、多随机种子)验证了该模式普遍性,同时指出ChestX 1-shot为弱边界案例。研究证明零样本提示质量无法完整代理适配锚点质量,强调应在适配边界两侧评估语言有效性。
原文摘要 · Abstract (English)
Language descriptions in source-free cross-domain few-shot learning (SF-CDFSL) are often selected according to zero-shot accuracy obtained with a frozen vision--language model. This paper asks whether that ranking remains valid after target-domain visual adaptation. Under a strictly paired protocol, we compare a generic class-name template with fixed detailed class descriptions before and after visual Low-Rank Adaptation (LoRA) on EuroSAT, CropDisease, ISIC, and ChestX. Let $\deltazero$ and $\deltalora$ denote the Detailed-minus-Base accuracy before and after adaptation, respectively. Two recurring regimes emerge. In \emph{semantic saturation}, $\deltazero>0$ but $0<\deltalora\ll\deltazero$: on EuroSAT and CropDisease, initial gains of 8.13--21.54 percentage points contract to 0.69--2.96 points after LoRA. In \emph{semantic emergence}, $\deltazero\leq0$ but $\deltalora>0$: on ISIC and ChestX, detailed descriptions become more useful only after the visual representation is updated. Training trajectories and sample-level decomposition show that saturation is driven mainly by Base-LoRA recovering errors already solved by detailed semantics, whereas emergence is associated with prediction turnover and newly formed Detailed-only correct decisions. Fixed-point-free shuffled-semantic controls, a second CLIP backbone, and multiple random seeds support the broad pattern while identifying ChestX 1-shot as a weak boundary case. These findings establish that zero-shot prompt quality is an incomplete proxy for adaptation-anchor quality and motivate evaluating language on both sides of the adaptation boundary.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。