语言模型可逆，能精确还原输入文本。
Language Models are Injective and Hence Invertible
- 证明了语言模型在初始化时即为单射，输入唯一对应输出。
- 百亿次测试未发现任何输入冲突，验证了理论结果。
- 提出SipIt算法，线性时间恢复原始文本，适合可解释性研究。
Transformer中的非线性激活和归一化通常被认为是非单射的，可能导致不同输入映射到相同输出，阻碍从模型表示中完全恢复输入。本文挑战这一观点：首先，数学证明了将离散输入序列映射为连续表示序列的语言模型是单射且无损的，该性质在初始化时即成立并贯穿训练过程；其次，通过在六种先进语言模型上进行百亿次碰撞测试，未观察到任何碰撞；第三，提出了SipIt算法，首次实现对隐藏激活值的可证明且高效地精确重构，具备线性时间复杂度，实证了语言模型的实际可逆性。本工作确立了单射性作为语言模型的基本可利用属性，对透明性、可解释性和安全部署具有直接意义。
原文摘要 · Abstract (English)
Transformer components such as non-linear activations and normalization are inherently non-injective, suggesting that different inputs could map to the same output and prevent exact recovery of the input from a model's representations. In this paper, we challenge this view. First, we prove mathematically that transformer language models mapping discrete input sequences to their corresponding sequence of continuous representations are injective and therefore lossless, a property established at initialization and preserved during training. Second, we confirm this result empirically through billions of collision tests on six state-of-the-art language models, and observe no collisions. Third, we operationalize injectivity: we introduce SipIt, the first algorithm that provably and efficiently reconstructs the exact input text from hidden activations, establishing linear-time guarantees and demonstrating exact invertibility in practice. Overall, our work establishes injectivity as a fundamental and exploitable property of language models, with direct implications for transparency, interpretability, and safe deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。