让大模型推理与强化学习协作，解决跨领域对话的长期规划难题
Bridging Reasoning and Action: Hybrid LLM-RL Framework for Efficient Cross-Domain Task-Oriented Dialogue
- 用大模型提取潜在约束，再通过双重检验验证其正确性
- 验证后的约束转为结构化状态表示，提升长程任务表现
- 适合研究对话系统、强化学习融合的开发者参考
跨领域任务导向对话需在隐含与显式可行性约束下进行推理，并规划多轮长周期动作。大语言模型（LLMs）可推断这些约束，但在长序列中不可靠；强化学习（RL）能优化长周期行为，却无法从原始对话中恢复约束。直接耦合二者易导致状态表示污染和策略误导。为此，我们提出验证型大模型知识增强的强化学习框架（VLK-RL），先由大模型生成候选约束，再通过双角色交叉检验抑制幻觉和跨轮不一致。经验证的约束被映射为符合本体的槽值表示，形成结构化、约束感知的状态，用于强化学习策略优化。多个基准测试表明，该方法显著提升泛化能力与鲁棒性，在长周期任务上优于强基线单模型。
原文摘要 · Abstract (English)
Cross-domain task-oriented dialogue requires reasoning over implicit and explicit feasibility constraints while planning long-horizon, multi-turn actions. Large language models (LLMs) can infer such constraints but are unreliable over long horizons, while Reinforcement learning (RL) optimizes long-horizon behavior yet cannot recover constraints from raw dialogue. Naively coupling LLMs with RL is therefore brittle: unverified or unstructured LLM outputs can corrupt state representations and misguide policy learning. Motivated by this, we propose Verified LLM-Knowledge empowered RL (VLK-RL), a hybrid framework that makes LLM-derived constraint reasoning usable for RL. VLK-RL first elicits candidate constraints with an LLM and then verifies them via a dual-role cross-examination procedure to suppress hallucinations and cross-turn inconsistencies. The verified constraints are mapped into ontology-aligned slot-value representations, yielding a structured, constraint-aware state for RL policy optimization. Experiments across multiple benchmarks demonstrate that VLK-RL significantly improves generalization and robustness, outperforming strong single-model baselines on long-horizon tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。