通过数学方法揭示Transformer输入空间的等价类，找到语义相似但形式不同的输入。
Unveiling Transformer Perception by Exploring Input Manifolds
- 基于雅可比矩阵和距离度量反向投影，构建输入空间的几何变形模型。
- 能找出使输出分类分布相同的输入，也能发现导致不同分类结果的输入。
- 适用于理解模型决策机制，适合研究AI可解释性与对抗样本生成。
本文提出一种通用方法，用于探索Transformer模型输入空间中的等价类。该方法基于扎实的数学理论，将Transformer各层视为对输入流形的连续变形。通过计算输出空间距离度量经模型雅可比矩阵反向投影后的拉回，并进行特征分解，我们能够重建输入空间中的等价类并实现跨类导航。该方法支持两种互补的探索过程：一是检索产生相同类别概率分布的输入实例，从而识别同一等价类内的元素；二是发现导致不同类别概率分布的实例，有效导向其他等价类。最后，我们通过将嵌入向量投影回人类可读格式，验证了所获实例的语义可解释性。
原文摘要 · Abstract (English)
This paper introduces a general method for the exploration of equivalence classes in the input space of Transformer models. The proposed approach is based on sound mathematical theory which describes the internal layers of a Transformer architecture as sequential deformations of the input manifold. Using eigendecomposition of the pullback of the distance metric defined on the output space through the Jacobian of the model, we are able to reconstruct equivalence classes in the input space and navigate across them. Our method enables two complementary exploration procedures: the first retrieves input instances that produce the same class probability distribution as the original instance-thus identifying elements within the same equivalence class-while the second discovers instances that yield a different class probability distribution, effectively navigating toward distinct equivalence classes. Finally, we demonstrate how the retrieved instances can be meaningfully interpreted by projecting their embeddings back into a human-readable format.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。