用汉字字形图加速中文语言模型训练，初期效果翻倍但最终精度相同
Hot-Start Chinese Language Modeling:Visual Glyphs Accelerate Sample-Efficient Learning
- 将汉字转为视觉图像输入，提前编码部首结构作为先验知识
- 首个训练周期内准确率提升至12.3%，是传统方法的两倍以上
- 适合追求快速原型验证或低资源场景的中文模型研究者
本文研究将中文字符以视觉字形图像形式输入，而非主流大模型使用的离散标记ID，探索其作为字符级语言建模的归纳偏置。核心发现揭示双重效应：视觉输入能显著实现热启动，首次训练周期（仅占总训练步数的0.4%）内准确率从5.8%提升至12.3%，提升超过一倍；但两种方法最终收敛至基本相同的准确率（39%）。该现象在8×8像素低分辨率、最多50%部分裁剪及110M至1.78B参数规模下均成立。我们识别出机制在于：字形渲染在训练前即把部首结构编码进嵌入空间（余弦相似度0.27，随机嵌入为0.002），从而加速对齐，但不提升最终容量。结果澄清了视觉表示作为归纳偏置在中文建模中的潜力与根本局限。
原文摘要 · Abstract (English)
In this work, we study whether rendering Chinese characters as visual glyph images, rather than discrete token IDs as mainstream LLMs do, providing an inductive bias for character-level language modeling. Our central finding gives a double-edged insight: visual inputs produce a pronounced hot-start effect, more than doubling early-stage accuracy within the first epoch (at 0.4% of total training steps) (12.3% visual inputs vs. 5.8% index-based baseline), yet both approaches converge to essentially identical final accuracy (39%). This pattern holds across resolutions as low as 8x8 pixels, partial cropping up to 50%, and model scales from 110M to 1.78B parameters. The mechanism we identify is that glyph rendering pre-encodes radical-based structure into embedding space before any training (cosine similarity 0.27 vs. 0.002 for random embeddings), enabling faster alignment but not higher final capacity. Our results clarify both the promise and fundamental limitation of visual representations as inductive biases for Chinese language modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。