用分步提示链提升学术报告生成的可靠性和质量
Prompt Chaining in Practice: A Case Study in Automated Scholarly Report Generation
- 采用多阶段提示链设计，分步完成复杂信息合成
- 成功率100%，优于单次提示的50%失败率，ROUGE-L F1达0.507
- 适合需要高可靠性的自动化文献综述场景
学术出版物的指数级增长要求自动化工具进行有效信息整合。然而，简单的单次提示方法常因可靠性与质量不足而难以胜任复杂合成任务。本文提出并实证评估了一种多阶段提示链方法，作为此类任务更可靠的架构模式。该方法应用于我们的系统 AI SciBrief，可自动生成学术摘要。我们开展对比实验，将提示链方法与经过精心优化的单次提示基线在教育领域的人工标准报告上进行评估。结果表明，提示链方法实现100%成功，而基线在50%的运行中失败；在质量上，新方法的ROUGE-L F1得分（0.507）优于基线（0.486），主要由更高精度驱动。结论是，提示链是一种更可靠、高效的工程方法，能显著降低单体提示固有的失败与不一致风险。
原文摘要 · Abstract (English)
The exponential growth of scholarly publications requires automated tools for effective information synthesis. However, simple, single-shot prompting methods often lack the reliability and quality required for complex synthesis tasks. This paper introduces and empirically evaluates a multi-stage prompt chaining methodology as a more reliable architectural pattern for such tasks. This approach is implemented in our system, AI SciBrief, which automatically generates scholarly digests. We conducted a comparative experiment, measuring the performance of our prompt chaining method against a carefully optimized single-shot baseline. Both systems were evaluated against a human-authored "gold standard" report for the "Education" domain. The results demonstrate a significant difference in reliability: our prompt chaining method achieved a 100% success rate, whereas the optimized baseline failed in 50% of its runs. In terms of quality, the proposed method also demonstrated a clear advantage, achieving a superior ROUGE-L F1-score (0.507 vs. 0.486), driven primarily by higher precision. We conclude that prompt chaining is a more dependable and effective engineering approach for complex, multi-step generative tasks, significantly mitigating the risks of failure and inconsistency inherent in monolithic prompts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。