通过探测大模型内部推理电路，高效筛选高质量数学训练数据
CircuitSeer: Mining High-Quality Data by Probing Mathematical Reasoning Circuits in LLMs
- 基于模型内部注意力头激活模式识别核心推理电路
- 用10%精选数据使模型性能比全量数据提升1.4分
- 适合追求高效微调的数学推理研究者
大语言模型在复杂推理任务上表现优异，但提升性能通常依赖大规模推理数据集，训练成本高昂。现有数据筛选方法多依赖外部模型或难以解释的启发式规则。本文转向模型内部机制，发现复杂推理任务会稳定激活一组稀疏且专门化的注意力头，形成核心推理电路。基于此，我们提出CircuitSeer，通过测量样本对这些关键电路的影响来量化其推理复杂度，实现数据筛选。在4个模型和9个数据集上的实验证明其优越性。值得注意的是，仅使用10%由CircuitSeer筛选的数据微调Qwen2.5-Math-7B模型，平均Pass@1得分较全量数据训练提升1.4分，凸显其高效与有效。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated impressive reasoning capabilities, but scaling their performance often relies on massive reasoning datasets that are computationally expensive to train on. Existing data selection methods aim to curate smaller, high-quality subsets but often rely on costly external models or opaque heuristics. In this work, we shift the focus from external heuristics to the model's internal mechanisms. We find that complex reasoning tasks consistently activate a sparse, specialized subset of attention heads, forming core reasoning circuits. Building on this insight, we propose CircuitSeer, a novel data selection method that quantifies the reasoning complexity of data by measuring its influence on these crucial circuits. Extensive experiments on 4 models and 9 datasets demonstrate CircuitSeer's superiority. Notably, fine-tuning Qwen2.5-Math-7B on just 10% of data selected by our method achieves a 1.4-point gain in average Pass@1 over training on the full dataset, highlighting its efficiency and effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。