演示标签的重复会严重破坏少样本分类准确率，模型只认出现过的词。
In-Context Fixation: When Demonstrated Labels Override Semantics in Few-Shot Classification
- 模型把演示中的标签当作唯一答案库，忽略语义合理性。
- 同质标签下准确率降至12%以下，多标签任务甚至归零。
- 适合研究提示学习机制或模型行为偏差的人阅读。
尽管随机演示标签对上下文学习影响甚微（Min et al., 2022），我们发现同质标签——即使语义合理——也会使六种模型（Pythia、Llama、Qwen；0.8B–8B）在四个任务上的准确率降至≤12%。根源在于标签位置的文本内容：模型将占据标签位置的标记视为完整答案词汇表，同质性是极端压缩情形。一项新的集合级固定发现证实：当演示使用{foo, bar, vex, nit, orb}等无意义词时，模型对这些展示集合的概率达42–67%，而“狗”的概率仍低于0.2%。这与潜在概念贝叶斯解释不一致，表明ICL输出实为受约束的词汇检索——模型输出绑定于演示中出现的词元，无视语义合理性。该效应扩展至四分类任务（三模型，1B–8B，准确率为0%）和多词表述（如“very positive”），并可分解为格式层（模板采纳）与内容层（极性覆盖）两部分，实验上可分离。机制层面，对Pythia-1B进行逐项配对激活修补恢复98.4%差距（95%置信区间[84%, 112%]），定位到第7层为中心的回路（秩2/560，99.8百分位；4次交叉验证均值103%）。跨架构的logit lens在Llama-3.2-1B上复现编码后覆盖轨迹，并通过因果验证确认（前5层贡献89%）。
原文摘要 · Abstract (English)
While random demonstration labels barely hurt in-context learning (Min et al., 2022), we show that homogeneous labels--even semantically valid ones--collapse accuracy to <=12% across six models (Pythia, Llama, Qwen; 0.8B--8B) and four tasks. The trigger is label-slot content: the model treats tokens occupying the label position as an exhaustive answer vocabulary, with homogeneity as the maximally collapsed case. A novel set-level fixation finding confirms this: when demonstrations carry varied nonsense tokens from {foo,bar,vex,nit,orb}, the model places 42--67% of probability on the demonstrated set while P(dog) remains below 0.2%. This is inconsistent with latent-concept Bayesian accounts (Xie et al., 2022) and reveals that ICL output is constrained vocabulary retrieval--the model binds its output to the demonstrated token inventory regardless of semantic plausibility. The effect generalizes to 4-way classification (0% accuracy across three models, 1B--8B) and multi-token verbalizers ("very positive"), where we decompose fixation into format-level (template adoption) and content-level (polarity override) components that are experimentally dissociable. Mechanistically, per-item paired activation patching on Pythia-1B recovers 98.4% of the gap (95% CI [84%, 112%]), localizing fixation to a layer-7-centered circuit (rank 2/560, 99.8th percentile; 4-fold CV mean 103%). Cross-architecture logit lens on Llama-3.2-1B replicates the encode-then-override trajectory with causal confirmation (top-5 layers: 89% recovery).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。