提出任务特异性分数，衡量指令对输出的决定性作用。
Task--Specificity Score: Measuring How Much Instructions Really Matter for Supervision
- 用真实指令与合理替代指令对比，量化指令重要性。
- 筛选高特异性指令可提升小模型在有限token下的表现。
- 适合关注指令质量与训练效率的研究者。
指令微调已成为大语言模型训练的标准方法，但许多指令-输入-输出对仅被弱指定：同一输入下，多个不同指令仍可能对应合理输出。这引出一个核心问题： 指令是否唯一决定了目标输出？为此，我们提出任务特异性分数（TSS），通过对比真实指令与同一输入下的合理替代指令，量化指令对输出预测的影响程度。我们进一步提出TSS++，采用硬性替代项并引入小质量修正项，以缓解易负样本带来的偏差。在三个指令数据集（Alpaca、Dolly-15k、NI-20）和三个开源大模型（Gemma、Llama、Qwen）上验证，选择高任务特异性示例能在严格令牌预算下提升下游性能，并与困惑度、IFD等质量过滤器形成互补。
原文摘要 · Abstract (English)
Instruction tuning is now the default way to train and adapt large language models, but many instruction--input--output pairs are only weakly specified: for a given input, the same output can remain plausible under several alternative instructions. This raises a simple question: \emph{does the instruction uniquely determine the target output?} We propose the \textbf{Task--Specificity Score (TSS)} to quantify how much an instruction matters for predicting its output, by contrasting the true instruction against plausible alternatives for the same input. We further introduce \textbf{TSS++}, which uses hard alternatives and a small quality term to mitigate easy-negative effects. Across three instruction datasets (\textsc{Alpaca}, \textsc{Dolly-15k}, \textsc{NI-20}) and three open LLMs (Gemma, Llama, Qwen), we show that selecting task-specific examples improves downstream performance under tight token budgets and complements quality-based filters such as perplexity and IFD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。