拆解大模型反事实推理过程,发现模态与中间推理影响性能关键
On the Eligibility of LLMs for Counterfactual Reasoning: A Decompositional Study
- 分阶段分析反事实生成:从因果构建到干预推理
- 跨多任务数据集验证,发现模态类型显著影响表现
- 为提升模型可靠性提供可解释框架,适合研究推理机制者
反事实推理已成为提升大语言模型推理泛化能力的关键技术。通过生成并分析反事实情境,研究人员可评估模型决策的适应性与可靠性。尽管已有研究表明大模型在反事实推理中常表现不佳,但导致其性能受限的核心因素在不同任务和模态下的作用仍不明确。本文提出一种分解式分析策略,将反事实生成过程拆解为因果构建与反事实干预推理两个阶段。为支持分解分析,我们考察了涵盖自然语言理解、数学、编程及视觉-语言任务的多个任务数据集。通过大规模评估,我们刻画了大模型在各阶段的行为特征,并揭示了模态类型与中间推理过程对性能的影响。本工作建立了一个结构化的反事实推理分析框架,推动更可靠的基于大模型的推理系统发展,并为未来推理激发策略提供指导。
原文摘要 · Abstract (English)
Counterfactual reasoning has emerged as a crucial technique for generalizing the reasoning capabilities of large language models (LLMs). By generating and analyzing counterfactual scenarios, researchers can assess the adaptability and reliability of model decision-making. Although prior work has shown that LLMs often struggle with counterfactual reasoning, it remains unclear which factors most significantly impede their performance across different tasks and modalities. In this paper, we propose a decompositional strategy that breaks down the counterfactual generation from causality construction to the reasoning over counterfactual interventions. To support decompositional analysis, we investigate \ntask datasets spanning diverse tasks, including natural language understanding, mathematics, programming, and vision-language tasks. Through extensive evaluations, we characterize LLM behavior across each decompositional stage and identify how modality type and intermediate reasoning influence performance. By establishing a structured framework for analyzing counterfactual reasoning, this work contributes to the development of more reliable LLM-based reasoning systems and informs future elicitation strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。