用塞尔维亚语双书写系统测试大模型是否真懂语义,发现模型不被字形迷惑。
One Language, Two Scripts: Probing Script-Invariance in LLM Concept Representations
- 用拉丁与西里尔字母写相同句子,看模型特征是否一致
- 不同字形下特征重合度远超随机水平,且比改写还高
- 适合研究模型抽象表征能力,尤其关注语义而非字形
稀疏自编码器(SAE)学习到的特征是表达抽象语义,还是依赖文字书写方式?我们以塞尔维亚语双书写系统为受控实验环境:塞尔维亚语可自由使用拉丁或西里尔字母,字符映射近乎完美,使语义恒定而字形变化。关键在于两种书写系统分词完全不同,无任何共享词元。分析Gemma模型系列(270M-27B参数)的SAE特征激活,发现同一句子在不同书写系统下激活高度重叠的特征,远超随机基线。令人惊讶的是,更换书写系统带来的表征差异,甚至小于同一书写系统内的改写。跨书写、跨改写组合虽极少在训练数据中共同出现,但仍表现出显著特征重合,排除了单纯记忆的可能。模型规模越大,这种书写不变性越强。结果表明,SAE特征能捕捉高于分词表面形式的语义抽象层次,并建议将塞尔维亚双书写系统作为通用评估范式,用于探测模型表征的抽象程度。
原文摘要 · Abstract (English)
Do the features learned by Sparse Autoencoders (SAEs) represent abstract meaning, or are they tied to how text is written? We investigate this question using Serbian digraphia as a controlled testbed: Serbian is written interchangeably in Latin and Cyrillic scripts with a near-perfect character mapping between them, enabling us to vary orthography while holding meaning exactly constant. Crucially, these scripts are tokenized completely differently, sharing no tokens whatsoever. Analyzing SAE feature activations across the Gemma model family (270M-27B parameters), we find that identical sentences in different Serbian scripts activate highly overlapping features, far exceeding random baselines. Strikingly, changing script causes less representational divergence than paraphrasing within the same script, suggesting SAE features prioritize meaning over orthographic form. Cross-script cross-paraphrase comparisons provide evidence against memorization, as these combinations rarely co-occur in training data yet still exhibit substantial feature overlap. This script invariance strengthens with model scale. Taken together, our findings suggest that SAE features can capture semantics at a level of abstraction above surface tokenization, and we propose Serbian digraphia as a general evaluation paradigm for probing the abstractness of learned representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。