用稀疏自编码器挖掘文本与目标变量的可解释关系，生成可理解的假设。
Sparse Autoencoders for Hypothesis Generation
- 训练稀疏自编码器提取可解释的文本特征
- 在真实数据上生成约两倍于基线的显著发现
- 计算量低，适合快速生成可解释性假设
我们提出HypotheSAEs，一种通用方法，用于从文本数据（如新闻标题）中推测其与目标变量（如点击率）之间的可解释关系。该方法包括三步：(1) 在文本嵌入上训练稀疏自编码器，生成描述数据分布的可解释特征；(2) 选取能预测目标变量的特征；(3) 利用大语言模型为每个特征生成自然语言解释（如“提及惊讶或震惊”），作为对目标变量的预测假设。相较于基线方法，该方法在合成数据上至少提升0.06的F1值，在真实数据上产生的可预测假设数量约为基线的两倍，且所需计算量仅为近期基于LLM方法的1-2个数量级。HypotheSAEs还在两个经典任务中发现新见解：解释国会演讲中的党派差异，以及识别影响在线标题传播的关键因素。
原文摘要 · Abstract (English)
We describe HypotheSAEs, a general method to hypothesize interpretable relationships between text data (e.g., headlines) and a target variable (e.g., clicks). HypotheSAEs has three steps: (1) train a sparse autoencoder on text embeddings to produce interpretable features describing the data distribution, (2) select features that predict the target variable, and (3) generate a natural language interpretation of each feature (e.g., "mentions being surprised or shocked") using an LLM. Each interpretation serves as a hypothesis about what predicts the target variable. Compared to baselines, our method better identifies reference hypotheses on synthetic datasets (at least +0.06 in F1) and produces more predictive hypotheses on real datasets (~twice as many significant findings), despite requiring 1-2 orders of magnitude less compute than recent LLM-based methods. HypotheSAEs also produces novel discoveries on two well-studied tasks: explaining partisan differences in Congressional speeches and identifying drivers of engagement with online headlines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。