让大模型像人一样思考规划,提升对话准确性与任务完成率。
LLM should think and action as a human
- 引入内置思维链,结合上下文与记忆进行推理和规划。
- 通过监督学习与强化学习双路径训练,显著提升推理与执行能力。
- 适合需要复杂任务处理的智能助手场景,如客服、编程辅助。
当前大语言模型多用于对话助理,但多轮对话中易出错且难以达成目标,错误概率随对话轮次增加而上升;同一提示下难以灵活生成不同响应流程;工具调用方式笨拙低效,调用次数受限。根源在于模型缺乏人类般的思考、推理与规划能力。为此,本文提出基于内置思维链的思考方法:在每轮对话中,模型综合聊天历史、思维上下文、行动调用、记忆与知识进行详细推理与计划,并按计划执行。通过该方法构建训练数据集,采用监督学习微调模型;同时训练一致性奖励模型,以强化学习进一步优化模型输出行为。实验表明,模型的推理与规划能力显著增强,多轮对话中的问题得到有效解决。
原文摘要 · Abstract (English)
It is popular lately to train large language models to be used as chat assistants, but in the conversation between the user and the chat assistant, there are prompts, require multi-turns between the chat assistant and the user. However, there are a number of issues with the multi-turns conversation: The response of the chat assistant is prone to errors and can't help users achieve their goals, and as the number of conversation turns increases, the probability of errors will also increase; It is difficult for chat assistant to generate responses with different processes based on actual needs for the same prompt; Chat assistant require the use of tools, but the current approach is not elegant and efficient, and the number of tool calls is limited. The main reason for these issues is that large language models don't have the thinking ability as a human, lack the reasoning ability and planning ability, and lack the ability to execute plans. To solve these issues, we propose a thinking method based on a built-in chain of thought: In the multi-turns conversation, for each user prompt, the large language model thinks based on elements such as chat history, thinking context, action calls, memory and knowledge, makes detailed reasoning and planning, and actions according to the plan. We also explored how the large language model enhances thinking ability through this thinking method: Collect training datasets according to the thinking method and fine tune the large language model through supervised learning; Train a consistency reward model and use it as a reward function to fine tune the large language model using reinforcement learning, and the reinforced large language model outputs according to this way of thinking. Our experimental results show that the reasoning ability and planning ability of the large language model are enhanced, and the issues in the multi-turns conversation are solved.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。