用自动生成的思维链减少小模型因果幻觉。
Generating Effective CoT Traces for Mitigating Causal Hallucination
- 设计生成符合标准的思维链流水线，针对性缓解小模型因果幻觉。
- 新提出因果幻觉率（CHR）指标，验证生成思维链有效降低幻觉。
- 生成的思维链让小模型在多数据集上表现更稳，抗干扰能力强。
尽管大语言模型在复杂推理任务中表现出色，但在事件因果识别（ECI）中，特别是参数量小于等于1.5B的小模型，仍存在严重的因果幻觉问题。一种有前景的解决方法是使用思维链（CoT）轨迹进行微调，但目前缺乏适用于ECI的CoT轨迹数据集。本文首先分析了有效CoT轨迹应具备的关键特征，以缓解小模型中的因果幻觉。随后设计了一条满足这些特征的轨迹生成流水线。此外，由于尚无量化因果幻觉的指标，本文还引入了新的因果幻觉率（Causal Hallucination Rate, CHR）来量化幻觉程度，指导有效轨迹标准的制定，并验证流水线的有效性。实验表明，使用本流水线生成的CoT轨迹对小模型进行微调，不仅显著降低了因果幻觉，还提升了平均准确率。同时，微调后的模型在跨数据集、跨难度场景下展现出强泛化能力，并对误导性提示具有鲁棒性。
原文摘要 · Abstract (English)
Although large language models (LLMs) excel in complex reasoning tasks, they suffer from severe causal hallucination in event causality identification (ECI), particularly in smaller models ($\leq$1.5B parameters). A promising approach to address this issue is to fine-tune them with Chain-of-Thought (CoT) traces. However, there is currently a lack of CoT trace dataset available for ECI. In this paper, we first investigate the essential criteria that effective CoT traces should possess to mitigate causal hallucination in smaller models. We then design a pipeline to generate CoT traces that meet these criteria. Moreover, since there is currently no metric for quantifying causal hallucination, we also introduce a new metric, the Causal Hallucination Rate (CHR), to quantify causal hallucination, guide the formulation of effective CoT trace criteria, and validate the effectiveness of our pipeline. Our experiments show that fine-tuning with the CoT traces generated by our pipeline not only substantially reduces causal hallucination in smaller LLMs but also improves mean accuracy. Moreover, the fine-tuned models exhibit strong cross-dataset and cross-difficulty generalization, as well as robustness under misleading intervention prompts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。