用小模型控制大模型，让黑盒LLM能自我优化生成复杂任务。
Matryoshka Pilot: Learning to Drive Black-Box LLMs with LLMs
- 用轻量级白盒模型分步引导黑盒LLM输出中间结果。
- 在多轮交互中调整提示词，使输出逐步对齐偏好。
- 适合无法微调的黑盒模型场景，提升长程任务能力。
尽管黑盒大型语言模型（LLMs）具备强大的生成能力，但其内在的不透明性阻碍了推理、规划和个人化等能力的进一步发展。现有方法通过领域特定适配来增强LLM能力，但需要在可访问的模型参数上进行额外训练，这对黑盒LLM不可行。为此，我们提出Matryoshka Pilot（M-Pilot），一个轻量级白盒LLM控制器，通过将复杂任务分解为一系列中间输出，引导大规模黑盒LLM生成器。具体地，我们将黑盒LLM视为环境，由M-Pilot作为策略，通过提示词提供中间指导以驱动黑盒LLM。M-Pilot通过迭代交互训练，调整黑盒LLM输出以对齐偏好，实现可控的多轮生成与中间引导的自我优化。在多样化任务上的实证评估表明，该方法有效提升了黑盒LLM在复杂、长时程任务中的能力。代码已公开于：https://github.com/lichangh20/Matryoshka。
原文摘要 · Abstract (English)
Despite the impressive generative abilities of black-box large language models (LLMs), their inherent opacity hinders further advancements in capabilities such as reasoning, planning, and personalization. Existing works aim to enhance LLM capabilities via domain-specific adaptation, which require additional training on accessible model parameters, an infeasible option for black-box LLMs. To address this challenge, we introduce Matryoshka Pilot (M-Pilot), a lightweight white-box LLM controller that guides a large-scale black-box LLM generator by decomposing complex tasks into a series of intermediate outputs. Specifically, we consider the black-box LLM as an environment, with M-Pilot serving as a policy to provide intermediate guidance through prompts for driving the black-box LLM. M-Pilot is trained to pivot the outputs of the black-box LLM aligning with preferences during iterative interaction, which enables controllable multi-turn generation and self-improvement in optimizing intermediate guidance. Empirical evaluations on diverse tasks demonstrate that our method effectively enhances the capabilities of black-box LLMs in complex, long-horizon tasks. Our code is publicly available at: https://github.com/lichangh20/Matryoshka.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。