解析分子Transformer的内部机制，揭示其生成合理分子的原理。
Circuits, Features, and Heuristics in Molecular Transformers
- 通过分析自回归Transformer，发现其具备语法解析与化学规则约束能力
- 利用稀疏自编码器提取出与化学特性相关的激活特征
- 成果可提升下游任务性能，适合药物设计研究者参考
Transformer能够生成有效且多样的化学结构，但其捕捉分子表示规则的内在机制仍不明确。本文对基于类药物小分子训练的自回归Transformer进行机制分析，揭示其在多个抽象层次上的计算结构。研究识别出与低级语法解析及更高级化学有效性约束一致的计算模式。通过稀疏自编码器（SAEs）提取与化学相关激活模式对应的特征字典，并在下游任务中验证了这些发现的有效性，表明机制洞察可转化为多种实际场景中的预测性能提升。
原文摘要 · Abstract (English)
Transformers generate valid and diverse chemical structures, but little is known about the mechanisms that enable these models to capture the rules of molecular representation. We present a mechanistic analysis of autoregressive transformers trained on drug-like small molecules to reveal the computational structure underlying their capabilities across multiple levels of abstraction. We identify computational patterns consistent with low-level syntactic parsing and more abstract chemical validity constraints. Using sparse autoencoders (SAEs), we extract feature dictionaries associated with chemically relevant activation patterns. We validate our findings on downstream tasks and find that mechanistic insights can translate to predictive performance in various practical settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。