不同大模型对少量示例的响应差异巨大，效果不随示例增多而提升。
When Does Few-Shot Prompting Help? A Systematic Empirical Study of Shot-Count Effects Across Model Scale, Architecture, and Output Parsing Robustness
- 控制变量实验对比5个模型在0到8个示例下的表现
- 部分模型用1个示例性能飙升，有的反而越加越差
- 发现并修正了影响评测结果的解析错误问题
少样本提示是自然语言处理中广泛使用的推理技术，但其与模型规模、架构和输出格式合规性的关系缺乏系统研究。本文在AG News四分类数据集（n=200）上，对五个模型（Gemini Flash Lite、GPT-4o-mini、Llama 3.1 8B、Llama 3.3 70B、Llama 4 Scout 17B）在六种示例数（k ∈ {0,1,2,3,5,8}）下进行受控实验。报告宏平均F1值、95%自助置信区间（B=10,000）、置换检验p值及Cohen's d效应量。结果显示四类行为模式：(1) 零样本已校准，少量示例无显著增益（Gemini、GPT-4o-mini）；(2) 零样本崩溃，单示例即大幅恢复（Llama 3.1 8B，d=10.98，p<0.0001）；(3) 零样本最优，后续示例导致性能持续下降（Llama 4 Scout）；(4) U型曲线（Llama 3.3 70B：0-shot F1=0.907，2-shot F1=0.635，5-shot F1=0.785，经解析修正后）。还识别并纠正了使Llama 3.3 70B性能被低估高达206%的系统性解析缺陷，对评估方法具有重要贡献。结果表明，示例数与性能的关系非单调、非普适，也无法仅凭模型规模预测。
原文摘要 · Abstract (English)
Few-shot prompting, the practice of prepending a small number of input-output demonstration pairs to a query before presenting it to a large language model (LLM), is among the most widely adopted inference-time techniques in NLP. Yet little systematic work investigates how shot count interacts with model scale, architecture, and output format compliance in determining classification performance. This paper presents a controlled study of five LLMs across six shot-count configurations (k in {0,1,2,3,5,8}) on the AG News four-class benchmark (n=200). Our models span proprietary and open-source families: Gemini Flash Lite, GPT-4o-mini, Llama 3.1 8B, Llama 3.3 70B, and Llama 4 Scout 17B. We report macro-averaged F1 with 95% bootstrap confidence intervals (B=10,000), permutation-test p-values, and Cohen's d effect sizes across all 30 configurations. Our findings reveal four qualitatively distinct behavioral regimes: (1) models already well-calibrated at zero-shot that show modest, statistically insignificant gains (Gemini, GPT-4o-mini); (2) models that undergo catastrophic zero-shot failure but recover dramatically with a single example (Llama 3.1 8B, d=10.98, p<0.0001); (3) models optimal at zero-shot that degrade monotonically with additional examples (Llama 4 Scout); and (4) models exhibiting a U-shaped curve (Llama 3.3 70B: 0-shot F1=0.907, 2-shot F1=0.635, 5-shot F1=0.785 with parser corrected). We additionally identify, diagnose, and correct a systematic parsing artifact that artificially deflated Llama 3.3 70B performance by up to 206%, constituting a methodological contribution to LLM evaluation practice. Our results demonstrate that the relationship between shot count and classification performance is not monotonic, not universal, and not predictable from model scale alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。