用强化搜索生成符合心理辅导原则的对话,让AI更懂共情。
MCTSr-Zero: Self-Reflective Psychological Counseling Dialogues Generation via Principles and Adaptive Exploration
- 以领域原则替代固定目标,引导对话向共情方向演进。
- 通过再生与元提示自适应,探索多样初始策略,提升对话质量。
- 专为心理辅导设计,适合需要高伦理与情感适配的场景。
将蒙特卡洛树搜索(MCTS)与大语言模型(LLMs)结合已在结构化任务中取得显著成效,但在开放式对话(如心理辅导)中面临独特挑战。与存在客观正确性的任务不同,治疗性对话的成功依赖于共情、伦理合规和人类偏好等主观因素,难以定义严格“正确”标准。现有结果导向的MCTS方法因此易产生不匹配回应。为此,我们提出MCTSr-Zero框架,专用于开放式的以人为本对话。其核心创新在于“领域对齐”,将搜索目标从预设终点转向符合目标领域原则(如辅导中的共情)的对话轨迹。此外,引入“再生”与“元提示自适应”机制,大幅拓宽探索空间,使MCTS可考虑根本不同的初始对话策略。我们在心理辅导场景生成多轮对话数据,用于微调语言模型PsyLLM。同时提出PsyEval基准,评估多轮心理辅导对话。实验表明,PsyLLM在PsyEval及其他相关指标上达到当前最优表现,验证了MCTSr-Zero在生成高质量、原则对齐对话数据方面的有效性,解决了大模型在复杂心理标准下一致性不足的问题。
原文摘要 · Abstract (English)
The integration of Monte Carlo Tree Search (MCTS) with Large Language Models (LLMs) has demonstrated significant success in structured, problem-oriented tasks. However, applying these methods to open-ended dialogues, such as those in psychological counseling, presents unique challenges. Unlike tasks with objective correctness, success in therapeutic conversations depends on subjective factors like empathetic engagement, ethical adherence, and alignment with human preferences, for which strict "correctness" criteria are ill-defined. Existing result-oriented MCTS approaches can therefore produce misaligned responses. To address this, we introduce MCTSr-Zero, an MCTS framework designed for open-ended, human-centric dialogues. Its core innovation is "domain alignment", which shifts the MCTS search objective from predefined end-states towards conversational trajectories that conform to target domain principles (e.g., empathy in counseling). Furthermore, MCTSr-Zero incorporates "Regeneration" and "Meta-Prompt Adaptation" mechanisms to substantially broaden exploration by allowing the MCTS to consider fundamentally different initial dialogue strategies. We evaluate MCTSr-Zero in psychological counseling by generating multi-turn dialogue data, which is used to fine-tune an LLM, PsyLLM. We also introduce PsyEval, a benchmark for assessing multi-turn psychological counseling dialogues. Experiments demonstrate that PsyLLM achieves state-of-the-art performance on PsyEval and other relevant metrics, validating MCTSr-Zero's effectiveness in generating high-quality, principle-aligned conversational data for human-centric domains and addressing the LLM challenge of consistently adhering to complex psychological standards.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。