用非周期拼图压缩大模型,分段问答微调,高效适配材料科学领域。
Penrose Tiled Low-Rank Compression and Section-Wise Q&A Fine-Tuning: A General Framework for Domain-Specific Large Language Model Adaptation
- 将权重矩阵分块成低秩块,按彭罗斯非周期拼图排列并压缩。
- 分段问答微调保留通用语言能力,有效注入材料科学知识。
- 适合数据稀缺领域的专业大模型定制,如材料科学、医学等。
大语言模型在材料科学等专业领域前景广阔,但受限于数据稀少和知识密集,高效准确的领域适配仍具挑战。本文提出两阶段框架:首先将模型权重矩阵分解为局部低秩“秩块”,以彭罗斯式非周期拼图方式排列,并通过离散余弦或傅里叶变换进行压缩,同时使用基于KL散度的对齐损失保持压缩后表示与原始模型分布一致。其次,在适配阶段采用类人类科研阅读流程:逐节处理材料科学文献,每节开展结构化问答,提取显式推理轨迹,逐步注入领域知识,同时最小化对通用语言能力的灾难性遗忘。该方法在数据稀缺条件下实现大模型的精准专业化,为材料科学知识融合提供可扩展的系统路径,未来工作将开展全面实证评估。
原文摘要 · Abstract (English)
Large language models (LLMs) hold great promise for specialized scientific domains such as materials science, yet adapting them efficiently and accurately to domain-specific knowledge remains challenging due to limited data and high knowledge density. We propose a two-stage framework that combines structured model compression with a scientific fine-tuning regimen to address this challenge. In the compression stage, we decompose the LLM's weight matrices into local low-rank "rank blocks" and arrange these blocks in a Penrose-like non-periodic tiling pattern. Each block is then compacted via spectral transformations (e.g., discrete cosine or Fourier transforms), and a Kullback-Leibler (KL) divergence-based alignment loss preserves the distributional similarity between the compressed model's representations and those of the original full model. In the adaptation stage, the compressed model is further tuned using a human-like scientific reading protocol: it processes technical materials science documents section by section, engaging in a structured question-and-answer routine for each section. This section-wise Q&A fine-tuning strategy extracts explicit reasoning traces and gradually injects domain knowledge, while minimizing catastrophic forgetting of the model's general language capabilities. By balancing efficient compression with targeted adaptation, our two-stage approach enables precise specialization of LLMs to high-value domains under data-scarce conditions. We present this principled yet exploratory pipeline and outline its potential for advancing materials science knowledge integration, laying the groundwork for comprehensive empirical evaluation in future work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。