arXiv:2601.09805cs.AIcs.LG2026-01Conference of the …被引 3

通过注意力干预提升大模型逻辑推理能力,无需外部工具。

Improving Chain-of-Thought for Logical Reasoning via Attention-Aware Intervention

  • 识别逻辑推理相关的注意力头并动态调整其权重。
  • 在多个基准上显著提升推理准确率,计算开销极低。
  • 适合需要高效可靠推理的部署场景,如智能问答系统。

当前大模型的逻辑推理主要依赖复杂的交互式框架,将推理过程分解为子任务,通过精心设计的提示或外部资源(如符号求解器)利用其强逻辑结构。然而,这类方法引入额外开销或依赖外部组件,限制了可扩展性。本文提出一种非交互式、端到端的推理框架,使推理能力在模型内部自然涌现,在保持可分析性的同时无需任何外部资源。我们发现,在少样本提示中引入结构信息会激活一组与逻辑推理算子模式对齐的注意力头。基于此,我们提出注意力感知干预(AAI),一种推理时的干预方法,通过重新加权由逻辑模式识别出的特定注意力头的注意力分数。AAI以高效方式引导模型通过注意力调制利用先验知识。大量实验表明,AAI在多种基准和模型架构上均显著提升逻辑推理性能,且计算开销可忽略不计。代码已公开于 https://github.com/phuongnm94/aai_for_logical_reasoning。

原文摘要 · Abstract (English)

Modern logical reasoning with LLMs primarily relies on employing complex interactive frameworks that decompose the reasoning process into subtasks solved through carefully designed prompts or requiring external resources (e.g., symbolic solvers) to exploit their strong logical structures. While interactive approaches introduce additional overhead or depend on external components, which limit their scalability. In this work, we introduce a non-interactive, end-to-end framework for reasoning tasks, enabling reasoning to emerge within the model itself-improving generalization while preserving analyzability without any external resources. We show that introducing structural information into the few-shot prompt activates a subset of attention heads that patterns aligned with logical reasoning operators. Building on this insight, we propose Attention-Aware Intervention (AAI), an inference-time intervention method that reweights attention scores across selected heads identified by their logical patterns. AAI offers an efficient way to steer the model's reasoning toward leveraging prior knowledge through attention modulation. Extensive experiments show that AAI enhances logical reasoning performance across diverse benchmarks, and model architectures, while incurring negligible additional computational overhead. Code is available at https://github.com/phuongnm94/aai_for_logical_reasoning.

逻辑推理注意力机制大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。