无需微调即可解释Transformer中的隐含语义，实现无监督分割
ULTra: Unveiling Latent Token Interpretability in Transformer-Based Understanding and Segmentation
- 通过无监督方法解析Transformer的隐含标记语义
- 在无监督分割任务上达到当前最优性能
- 适用于模型解释、物体选择与大模型摘要等场景
Transformer通过自注意力机制革新了计算机视觉，但其复杂性导致隐含标记表示难以解释。我们提出ULTra框架，用于解析Transformer嵌入并揭示其中的有意义语义模式。ULTra可在不进行微调的情况下，利用预训练模型实现无监督语义分割。此外，我们提出一种自监督训练方法，通过学习外部变换矩阵来提升分割性能，而无需修改底层模型。该方法在无监督语义分割任务上达到当前最优表现。同时，我们在合成数据和真实场景中验证了ULTra在模型解释方面的有效性，包括物体选择和基于大语言模型的可解释文本摘要，证明其在揭示隐含标记语义结构方面的广泛适用性。
原文摘要 · Abstract (English)
Transformers have revolutionized Computer Vision (CV) through self-attention mechanisms. However, their complexity makes latent token representations difficult to interpret. We introduce ULTra, a framework for interpreting Transformer embeddings and uncovering meaningful semantic patterns within them. ULTra enables unsupervised semantic segmentation using pre-trained models without requiring fine-tuning. Additionally, we propose a self-supervised training approach that refines segmentation performance by learning an external transformation matrix without modifying the underlying model. Our method achieves state-of-the-art performance in unsupervised semantic segmentation, outperforming existing segmentation methods. Furthermore, we validate ULTra for model interpretation on both synthetic and real-world scenarios, including Object Selection and interpretable text summarization using LLMs, demonstrating its broad applicability in explaining the semantic structure of latent token representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。