发现大模型内部存在跨格式的通用推理空间，能统一处理文字、代码和数学表达。
Beyond Language: Format-Agnostic Reasoning Subspaces in Large Language Models
- 通过多形式对比实验，找到模型中间层中共享的10维推理子空间。
- 仅替换这10维即可保留90%-96%输出准确率，远超传统方法。
- 揭示语言与形式差异背后的本质：陈述性与过程性表达不兼容。
大语言模型以不同表面形式（如英文段落、Python代码、数学符号）表达相同推理，但其内部是否共享共通表征尚不清楚。我们构建了包含324个样本的TriForm基准（18个概念×6种形式×3个实例），研究了五种规模为1.6B-8B的LLM（涵盖三种架构）。通过置换校正的RSA、跨格式探测和激活修补等方法，发现中层存在格式无关的推理子空间（FARS）。具体地，以概念中心点进行主成分分析，提取出一个10维子空间，使概念结构增强3倍，同时将形式信息抑制至接近零。在跨形式修补中仅替换这10维，便能保持90%-96%的模型输出性能——显著优于全激活替换（44%-56%）和方差最大化PCA（60%-74%）；删除这些维度则导致特定功能破坏。该子空间可泛化至未见概念，并在不同架构间收敛（所有模型对的CCA > 0.79），为柏拉图式表征假说提供模态内证据。进一步发现陈述性与过程性表达之间的不对称性：自然语言与数学表达的表征兼容性远高于与代码之间的兼容性，表明关键分歧并非语言与形式之别，而是陈述性与过程性之别。
原文摘要 · Abstract (English)
Large language models represent the same reasoning in vastly different surface forms -- English prose, Python code, mathematical notation -- yet whether they share a common internal substrate across these symbolic systems remains unknown. We introduce the TriForm Benchmark (18 concepts x 6 forms x 3 instances = 324 stimuli) and study five LLMs (1.6B-8B) across three architecture families. Using permutation-corrected RSA, cross-form probing, and activation patching, we find converging evidence for a Format-Agnostic Reasoning Subspace (FARS) in middle layers. We make FARS concrete: concept-centroid PCA extracts a 10-dimensional subspace that amplifies concept structure 3x while suppressing form information to near zero. Replacing only these 10 dimensions during cross-form patching preserves 90-96% of model output -- far exceeding both full activation replacement (44-56%) and variance-maximizing PCA (60-74%) -- while ablating them causes targeted disruption. FARS generalizes to held-out concepts and converges across architectures (CCA > 0.79 for all model pairs), providing within-modality evidence for the Platonic Representation Hypothesis. We further discover a declarative-procedural asymmetry: representations are far more compatible between prose and mathematics than between either and code, suggesting that the critical axis of divergence is not linguistic vs. formal but declarative vs. procedural.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。