arXiv:2605.18512cs.CL2026-05

通过判断演示效果替代盲目寻找,显著提升提示工程效率

Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selection

论文配图:Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selection
图 1 · 摘自论文原文
  • 用随机试验+轻量路由预测查询难度,分层评估演示效果
  • 在5个数据集上平均准确率领先基线3.4%,速度提升23倍
  • 适合需要快速验证提示质量的开发场景,尤其适合大模型应用

上下文学习(ICL)对提示中的示范样本极为敏感,但选择最优示范组合代价高昂,因可能组合空间巨大。本文提出核心观点:判断某查询-上下文对 $(q,D)$ 是否成功,比找到最优 $D^/star$ 更容易且更通用。基于此,提出 DiSP 框架——通过随机试验估算每个训练查询的成功率,训练轻量级路由模型根据查询特征预测难度,并为不同难度层级训练专属判别器。推理时,DiSP 在显式预算下执行‘满足即停’判断,若无合适上下文则输出诊断风险标签。在 Llama~3--8B 和 Qwen~2.5--7B 模型上,覆盖五个分类数据集,DiSP 实现最高平均准确率,较强的有学习选择基线提升达 3.4\%,同时实现最高 23× 的端到端运行时加速。

原文摘要 · Abstract (English)

In-context learning (ICL) is highly sensitive to which demonstrations appear in the prompt, but selecting them is expensive because the space of possible demonstration contexts and combinations is enormous. We argue that demonstration selection is \emph{easier to judge than to find}: predicting whether a specific query--context pair $(q,D)$ will succeed is cheaper and more general than searching for an optimal $D^\star$. Based on this insight, we propose DiSP, a sample-and-judge framework that stratifies queries by difficulty. DiSP runs random demonstration trials to estimate success rate of each training query, trains a lightweight router to predict difficulty from the query, and trains level-specific judges for sampled demonstrations. At inference, DiSP performs stop-on-acceptance judging under an explicit budget, emitting diagnostic risk tags when no suitable context is found. Across five classification datasets with Llama~3--8B and Qwen~2.5--7B, DiSP achieves the best average accuracy, improving over strong learned selection baselines by up to 3.4\%, while achieving up to $23\times$ end-to-end wall-clock speedup.

提示工程上下文学习模型效率智能选样

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。