arXiv:2409.19680cs.CLcs.AI2024-09NeurIPS被引 5

让大模型更懂指令背后的任务类型,提升任务识别准确率。

Instruction Embedding: Latent Representations of Instructions Towards Task Identification

  • 用提示工程提取聚焦任务的指令嵌入表示
  • 在IEB基准上任务分类准确率显著优于传统方法
  • 适合需要精准理解指令任务的下游应用

指令数据对提升大语言模型的人类级表现至关重要。近期研究LIMA表明,对齐本质是模型适应指令交互风格以解决各类任务的过程,依赖预训练知识与技能。因此,指令数据的核心在于其所代表的任务,而非具体语义或知识内容。指令的潜在表示在数据选择、示范检索等任务中起关键作用,但现有方法多基于文本嵌入,包含整体语义信息,干扰任务类别表征。本文提出新概念‘指令嵌入’,构建指令嵌入基准(IEB)用于训练与评估,并提出基线方法提示式指令嵌入(PIE),使表示更关注任务本身。在IEB上针对两个设计任务的评估显示,PIE在任务类别识别上表现更优。此外,指令嵌入在四个下游任务中的应用也验证了其有效性和适用性。

原文摘要 · Abstract (English)

Instruction data is crucial for improving the capability of Large Language Models (LLMs) to align with human-level performance. Recent research LIMA demonstrates that alignment is essentially a process where the model adapts instructions' interaction style or format to solve various tasks, leveraging pre-trained knowledge and skills. Therefore, for instructional data, the most important aspect is the task it represents, rather than the specific semantics and knowledge information. The latent representations of instructions play roles for some instruction-related tasks like data selection and demonstrations retrieval. However, they are always derived from text embeddings, encompass overall semantic information that influences the representation of task categories. In this work, we introduce a new concept, instruction embedding, and construct Instruction Embedding Benchmark (IEB) for its training and evaluation. Then, we propose a baseline Prompt-based Instruction Embedding (PIE) method to make the representations more attention on tasks. The evaluation of PIE, alongside other embedding methods on IEB with two designed tasks, demonstrates its superior performance in accurately identifying task categories. Moreover, the application of instruction embeddings in four downstream tasks showcases its effectiveness and suitability for instruction-related tasks.

指令嵌入任务识别大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。