用验证器强制纠错，让大模型规划结果100%正确
Robust Planning with Compound LLM Architectures: An LLM-Modulo Approach
- 大模型输出由多个可靠验证器实时校验，失败则自动重试
- 在4个调度任务中表现显著优于传统提示工程方法
- 适合对可靠性要求极高的自动化决策场景
以往工作通过提示工程提升大语言模型在规划与调度任务中的表现，但这些方法在测试分布内有效，缺乏鲁棒性和可预测性。本文提出一种复合大模型架构——LLM-Modulo框架，将大模型与一组完备的可靠验证器结合，对输出进行实时验证，若不通过则重新提示。该机制确保系统永不输出错误结果，实现输出绝对正确。在四个调度领域上的实验表明，该框架在不同模型上均取得显著性能提升。同时研究了框架基础配置的修改及其对整体性能的影响。
原文摘要 · Abstract (English)
Previous work has attempted to boost Large Language Model (LLM) performance on planning and scheduling tasks through a variety of prompt engineering techniques. While these methods can work within the distributions tested, they are neither robust nor predictable. This limitation can be addressed through compound LLM architectures where LLMs work in conjunction with other components to ensure reliability. In this paper, we present a technical evaluation of a compound LLM architecture--the LLM-Modulo framework. In this framework, an LLM is paired with a complete set of sound verifiers that validate its output, re-prompting it if it fails. This approach ensures that the system can never output any fallacious output, and therefore that every output generated is guaranteed correct--something previous techniques have not been able to claim. Our results, evaluated across four scheduling domains, demonstrate significant performance gains with the LLM-Modulo framework using various models. Additionally, we explore modifications to the base configuration of the framework and assess their impact on overall system performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。