小模型无法通过提示翻转标签语义,说明预训练时的语义锚点具有强约束力。
Semantic Anchors in In-Context Learning: Why Small LLMs Cannot Flip Their Labels
- 将大模型视为提示诱导分类器,对比自然与反转提示下的表现差异。
- 在8个任务和8个1-12B参数模型中,翻转标签后正确率始终为0。
- 揭示了少样本提示的根本局限:模型依赖预训练语义方向而非灵活重映射标签。
本文探讨大语言模型在上下文学习(ICL)中是否能覆盖预训练的标签语义,或仅是对已有语义框架的微调。通过将模型视为提示诱导分类器,对比自然演示(正确标签)与反转演示(系统性翻转标签含义)的行为,我们分解出三种对齐度量(真实性、先验与提示对齐),并引入语义覆盖率(正确性在翻转语义下的表现)。在八个分类任务和八个开源模型(1–12B参数)上,结果一致支持语义锚定观点:自然演示下,模型准确率提升且先验对齐强,多数正确预测与零样本行为一致,即使先验较弱;反转演示下,模型无法构建连贯的反语义分类器,提示对齐虽上升但准确性下降,且在1–12B参数设置下语义覆盖率为0。这表明ICL主要是在预训练阶段学到的稳定语义方向上调整输入投影,而非灵活重映射标签语义。这一发现揭示了少样本提示的根本局限,提示在当前规模下,要突破语义锚定需超越ICL的干预方式。所有代码已公开于:https://github.com/AnanthaPadmanaban-KrishnaKumar/semantic-anchors-icl。
原文摘要 · Abstract (English)
Can in-context learning (ICL) override pre-trained label semantics, or does it merely refine an existing semantic backbone? We address this question by treating LLMs as prompt-induced classifiers and contrasting their behavior under \emph{natural} demonstrations (with correct labels) and \emph{inverted} demonstrations (systematically flipping label meanings). We decompose ICL behavior into three alignment metrics (truth, prior, and prompt alignment) and introduce a semantic override rate, defined as correctness under flipped semantics. Across eight classification tasks and eight open-source LLMs (1--12B parameters), we find consistent evidence for a semantic anchor view. With natural demonstrations, ICL improves accuracy while maintaining strong prior alignment; most correct predictions coincide with zero-shot behavior, even when the prior is weak. With inverted demonstrations, models cannot learn coherent anti-semantic classifiers: prompt alignment increases only by sacrificing accuracy, and semantic override rates remain exactly zero in our few-shot 1--12B setting. Rather than flexibly remapping label meanings, ICL primarily adjusts how inputs project onto stable semantic directions learned during pre-training, clarifying fundamental limits of few-shot prompting and suggesting that overriding label semantics at these scales requires interventions beyond ICL. All code is available at: https://github.com/AnanthaPadmanaban-KrishnaKumar/semantic-anchors-icl.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。