arXiv:2604.00923cs.CL2026-04AAAI被引 1

揭示大模型学语言时感知与生成能力的分层机制,实现高效低资源适配。

Positional Cognitive Specialization: Where Do LLMs Learn To Comprehend and Speak Your Language?

  • 从输入和输出方向逐层消融,发现模型感知与生成能力在不同层分化。
  • 仅微调25%最外层(前后各1/4)即可接近全量微调效果,误差仅2-3%。
  • 提出CogSym方法,适合低资源语言快速适配,优于传统全量微调。

将大语言模型(LLMs)适配至新语言成本高且过程不透明。理解模型如何在训练中习得语言能力,是实现高效适配的关键。现有研究多关注已训练模型对多语言指令的处理,忽视了训练过程中语言习得机制。本文通过解码器仅结构的Transformer模型,从语言感知(输入理解)与产出(输出生成)两种认知功能专化角度,研究低资源语言下的训练动态。通过从输入端和输出端进行层消融实验,发现感知与生成能力在模型不同层中逐渐分化。基于此,提出一种分层启发式方法CogSym,仅微调少数早期与晚期层即可实现高效适配。结果表明,仅微调25%最外层,下游任务性能与全量微调相比偏差仅为2-3%。CogSym表现稳定,媲美LoRA等适配器方法,展现出超越全量微调的泛化能力。研究为理解大模型学语言机制提供了新视角,推动更可及、包容的语言建模发展。

原文摘要 · Abstract (English)

Adapting large language models (LLMs) to new languages is an expensive and opaque process. Understanding how language models acquire new languages and multilingual abilities is key to achieve efficient adaptation. Prior work on multilingual interpretability research focuses primarily on how trained models process multilingual instructions, leaving unexplored the mechanisms through which they acquire new languages during training. We investigate these training dynamics on decoder-only transformers through the lens of two functional cognitive specializations: language perception (input comprehension) and production (output generation). Through experiments on low-resource languages, we demonstrate how perceptual and productive specialization emerges in different regions of a language model by running layer ablation sweeps from the model's input and output directions. Based on the observed specialization patterns, we propose CogSym, a layer-wise heuristic that enables effective adaptation by exclusively fine-tuning a few early and late layers. We show that tuning only the 25% outermost layers achieves downstream task performance within 2-3% deviation from the full fine-tuning baseline. CogSym yields consistent performance with adapter methods such as LoRA, showcasing generalization beyond full fine-tuning. These findings provide insights to better understand how LLMs learn new languages and push toward accessible and inclusive language modeling.

大模型多语言低资源微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。