用模型内部特征指导数据选择，用一半数据达到全量训练效果
From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models
- 通过特征频率与干预筛选定位任务关键特征
- 数学任务上仅用50%数据提升17.4%性能，超越全量训练
- 适合追求高效微调的LLM研究者与工程师
尽管机制可解释性工具如稀疏自编码器（SAEs）能揭示大型语言模型（LLMs）中的有意义特征，但如何将这些洞察转化为实际优化策略仍是一大挑战。本文提出可解释性引导的数据选择（IGDS）框架，首先通过频率召回和干预过滤识别因果任务特征，再选取能最大激活这些特征的‘特征共振数据’用于微调。我们在Gemma-2、LLaMA-3.1和Qwen3模型的数学推理、摘要生成与翻译任务上验证了该方法。实验显示惊人数据效率：在数学任务中，IGDS在Gemma-2-2B上比全数据微调提升17.4%，仅使用50%数据，且优于关注数据质量与多样性的主流基线。分析表明特征增强与任务性能提升呈强正相关。IGDS为利用模型内生机制有效提升LLMs提供了直接而有效的路径，验证了核心假设。
原文摘要 · Abstract (English)
While mechanistic interpretability tools like Sparse Autoencoders (SAEs) can uncover meaningful features within Large Language Models (LLMs), a critical gap remains in transforming these insights into practical actions for model optimization. We bridge this gap with the hypothesis that data selection guided by a model's internal task features is a effective training strategy. Inspired by this, we propose Interpretability-Guided Data Selection (IGDS), a framework that first identifies these causal task features through frequency recall and interventional filtering, then selects ``Feature-Resonant Data'' that maximally activates task features for fine-tuning. We validate IGDS on mathematical reasoning, summarization, and translation tasks within Gemma-2, LLaMA-3.1, and Qwen3 models. Our experiments demonstrate exceptional data efficiency: on the Math task, IGDS surpasses full-dataset fine-tuning by a remarkable 17.4% on Gemma-2-2B while using only 50% of the data, and outperforms established baselines focused on data quality and diversity. Analysis confirms a strong positive correlation between feature amplification and task performance improvement. IGDS thus provides a direct and effective framework to enhance LLMs by leveraging their internal mechanisms, validating our core hypothesis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。