让大模型在不丢失指令能力的前提下，自适应新领域数据。
IKnow: Instruction-Knowledge-Aware Continual Pretraining for Effective Domain Adaptation
- 用对话式自监督目标训练模型，保留指令理解能力。
- 仅靠文本内隐含知识，无需外部数据库或原始模型权重。
- 适合模型权重受限或无专业语料的领域适配场景。
持续预训练有望仅使用无标注的测试时数据，将大语言模型（LLMs）适应到新领域，但直接对指令微调过的模型应用标准自监督目标，会损害其指令遵循能力和语义表征。现有方法通常需要访问原始基础模型或依赖外部领域专用数据库，这在模型权重因安全原因被隐藏或可靠外部语料不可用的场景中构成现实障碍。本文提出指令-知识感知持续适配框架（IKnow），通过在指令-响应对话格式中设计新颖的自监督目标，不依赖外部资源，而是利用文本自身蕴含的领域知识，并在更深层次的语义层面学习编码这些知识。
原文摘要 · Abstract (English)
Continual pretraining promises to adapt large language models (LLMs) to new domains using only unlabeled test-time data, but naively applying standard self-supervised objectives to instruction-tuned models is known to degrade their instruction-following capability and semantic representations. Existing fixes assume access to the original base model or rely on knowledge from an external domain-specific database - both of which pose a realistic barrier in settings where the base model weights are withheld for safety reasons or reliable external corpora are unavailable. In this work, we propose Instruction-Knowledge-Aware Continual Adaptation (IKnow), a simple and general framework that formulates novel self-supervised objectives in the instruction-response dialogue format. Rather than depend- ing on external resources, IKnow leverages domain knowledge embedded within the text itself and learns to encode it at a deeper semantic level.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。