用线性投影让不同模型的内部表示对齐,无需训练就能纠正推理错误。
Thinking in Different Spaces: Domain-Specific Latent Geometry Survives Cross-Architecture Translation
- 通过线性映射将大模型激活向量转到小模型坐标系中
- 跨20组异构模型对,推理准确率提升14%至50%
- 发现不同任务下模型敏感度相反,证明表征空间具有领域特异性
我们研究独立训练的语言模型是否趋向几何兼容的隐空间表示,并探索能否在不更新权重的情况下,通过推理时干预纠正模型行为。通过学习一个线性投影矩阵,将大型教师模型的激活向量映射到小型学生模型的坐标系中,在生成过程中替换学生模型残差流中的内部状态。在涵盖混合专家、密集、代码专用及合成训练架构的20组异构师生对上,岭回归投影在语言推理上实现R²=0.50,数学推理R²=0.40;而置换控制下降至R²=-0.22,L₁正则化下为R²=0.01。TruthfulQA行为纠正率在14.0%至50.0%之间(均值25.2%),GSM8K算术推理纠正率为8.5%至43.3%(均值25.5%),表明方法在不同推理领域具泛化能力。几何对齐质量与行为纠正率相关性极低(r = -0.07),揭示表征空间保真度与输出影响间存在解耦。干预强度具有架构特异性:最易操控的口语学生在数学任务中反而最难操控。20组对的双分离实验显示,投影矩阵在跨推理领域迁移时全部崩溃(双向平均R²=-3.83),确立了语言模型中领域特异性子空间几何为普遍特性。
原文摘要 · Abstract (English)
We investigate whether independently trained language models converge to geometrically compatible latent representations, and whether this compatibility can be exploited to correct model behavior at inference time without any weight updates. We learn a linear projection matrix that maps activation vectors from a large teacher model into the coordinate system of a smaller student model, then intervene on the student's residual stream during generation by substituting its internal state with the translated teacher representation. Across a fully crossed experimental matrix of 20 heterogeneous teacher-student pairings spanning mixture-of-experts, dense, code-specialized, and synthetically trained architectures, the Ridge projection consistently achieves R^2 = 0.50 on verbal reasoning and R^2 = 0.40 on mathematical reasoning, collapsing to R^2 = -0.22 under permutation control and R^2 = 0.01 under L_1 regularization. Behavioral correction rates range from 14.0% to 50.0% on TruthfulQA (mean 25.2%) and from 8.5% to 43.3% on GSM8K arithmetic reasoning (mean 25.5%), demonstrating that the method generalizes across fundamentally different reasoning domains. We report a near-zero correlation between geometric alignment quality and behavioral correction rate (r = -0.07), revealing a dissociation between representation space fidelity and output space impact. Intervention strength is architecture-specific: student models exhibit characteristic sensitivity profiles that invert across domains, with the most steerable verbal student becoming the least steerable mathematical student. Finally, a double dissociation experiment conducted across all 20 model pairings confirms without exception that projection matrices collapse catastrophically when transferred across reasoning domains (mean R^2 = -3.83 in both transfer directions), establishing domain-specific subspace geometry as a universal property of LMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。