揭示大模型语义空间中维诺图的几何特性，提出无需重训练的优化方法。
Geometric Properties of the Voronoi Tessellation in Latent Semantic Manifolds of Large Language Models
- 通过修正量化误差，验证了表达力差距的线性扩展规律。
- 发现中间层存在几何模糊区，最终层实现对齐，边际增益达28%。
- 基于费舍尔信息距离的优化可避免损伤，适合高精度微调场景。
语言模型在离散标记上运作却在连续向量空间中计算，从而在表征流形上诱导出维诺图划分。我们在Qwen3.5-4B-Base上进行实证研究，提出两项贡献：首先，采用float32边距重计算以消除bfloat16量化伪影，验证了Mabrok（2026）的表达力差距线性缩放律，$R^2$ = 0.9997——迄今最强证据；并识别出中层几何模糊区（第24-28层，$ρ$ = -0.29），其边距几何与交叉熵反相关，最终层趋于对齐（$ρ$ = 0.836）。其次，我们证明收敛模型的维诺图可通过边距精炼程序（MRP）重构：仅需短时后处理优化即可扩大标记决策边界，无需重新训练。对比直接边距最大化与费舍尔信息距离最大化的剂量响应实验，两者均达到约16,300个可修正位置/256K评估值上限，但损害机制不同。边距最大化损伤随干预强度上升，直至纠正失效；而费舍尔损伤稳定在约5,300位置（$λ$ = 0.15–0.6），在$λ$ = 0.6时实现+28%中位边距提升且下游基准不变，实现了表达力差距压缩与缩放律保持的几何重组。频率与标记类别审计显示，收益集中于高频结构标记（$λ$ = 0.6时占净修正84%），内容与实体类标记贡献随$λ$升高下降。因此，费舍尔MRP是一种可行的几何打磨工具，其实际上限由标记层面收益均匀性决定。
原文摘要 · Abstract (English)
Language models operate on discrete tokens but compute in continuous vector spaces, inducing a Voronoi tessellation over the representation manifold. We study this tessellation empirically on Qwen3.5-4B-Base, making two contributions. First, using float32 margin recomputation to resolve bfloat16 quantization artifacts, we validate Mabrok's (2026) linear scaling law of the expressibility gap with $R^2$ = 0.9997 - the strongest confirmation to date - and identify a mid-layer geometric ambiguity regime where margin geometry is anti-correlated with cross-entropy (layers 24-28, $ρ$ = -0.29) before crystallizing into alignment at the final layer ($ρ$ = 0.836). Second, we show that the Voronoi tessellation of a converged model is reshapable through margin refinement procedures (MRP): short post-hoc optimization runs that widen token-decision margins without retraining. We compare direct margin maximization against Fisher information distance maximization across a dose-response sweep. Both methods find the same ceiling of ~16,300 correctable positions per 256K evaluated, but differ critically in collateral damage. Margin maximization damage escalates with intervention strength until corrections are overwhelmed. Fisher damage remains constant at ~5,300 positions across the validated range ($λ$ = 0.15-0.6), achieving +28% median margin improvement at $λ$ = 0.6 with invariant downstream benchmarks - a geometric reorganization that compresses the expressibility gap while preserving its scaling law. However, frequency and token-class audits reveal that gains concentrate in high-frequency structural tokens (84% of net corrections at $λ$ = 0.6), with content and entity-like contributions shrinking at higher $λ$. Fisher MRP is therefore a viable geometric polishing tool whose practical ceiling is set not by aggregate damage but by the uniformity of token-level benefit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。