用自动生成脚本提升大模型任务提示效果,无需人工设计模板。
Task-oriented Prompt Enhancement via Script Generation
- 通过回溯提问和思维链生成任务步骤,自动构造执行脚本。
- 在11个任务中8次达到顶尖水平,比GPT-3.5和GPT-4的零样本方法分别高7.6%和3.9%。
- 无需人工标注,适合日常任务自动化,尤其适合快速部署场景。
大型语言模型(LLMs)在多种任务中展现出强大能力,但面对任务导向提示时因缺乏任务答案的先验知识而表现不佳。现有最优方法PAL依赖手工设计的提示模板与示例,仍存在结果不准确的问题。本文提出TITAN——一种新颖策略,通过通用化零样本学习生成脚本,显著提升LLMs在任务导向提示上的表现。TITAN无需详细任务特定指令或大量人工投入,利用两个关键机制:(1) 回溯提示提取任务输入规范,(2) 思维链提示识别所需操作步骤,以此优化代码生成过程。生成脚本经后处理并执行以获得最终答案。全面评估表明,TITAN平均优于当前最佳零样本方法7.6%(GPT-3.5)和3.9%(GPT-4);在11项任务中,无须人工标注即在8项达到顶尖性能,仅在3项略逊于需人工干预的少样本方法,差距微小。该工作为有效利用大模型完成日常任务提供了新范式。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable abilities across various tasks, leveraging advanced reasoning. Yet, they struggle with task-oriented prompts due to a lack of specific prior knowledge of the task answers. The current state-of-the-art approach, PAL, utilizes code generation to address this issue. However, PAL depends on manually crafted prompt templates and examples while still producing inaccurate results. In this work, we present TITAN-a novel strategy designed to enhance LLMs' performance on task-oriented prompts. TITAN achieves this by generating scripts using a universal approach and zero-shot learning. Unlike existing methods, TITAN eliminates the need for detailed task-specific instructions and extensive manual efforts. TITAN enhances LLMs' performance on various tasks by utilizing their analytical and code-generation capabilities in a streamlined process. TITAN employs two key techniques: (1) step-back prompting to extract the task's input specifications and (2) chain-of-thought prompting to identify required procedural steps. This information is used to improve the LLMs' code-generation process. TITAN further refines the generated script through post-processing and the script is executed to retrieve the final answer. Our comprehensive evaluation demonstrates TITAN's effectiveness in a diverse set of tasks. On average, TITAN outperforms the state-of-the-art zero-shot approach by 7.6% and 3.9% when paired with GPT-3.5 and GPT-4. Overall, without human annotation, TITAN achieves state-of-the-art performance in 8 out of 11 cases while only marginally losing to few-shot approaches (which needed human intervention) on three occasions by small margins. This work represents a significant advancement in addressing task-oriented prompts, offering a novel solution for effectively utilizing LLMs in everyday life tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。