发现大模型能识别输入领域并自动匹配最适配的模型
Exploring How LLMs Capture and Represent Domain-Specific Knowledge
- 通过预填充阶段隐藏状态分析模型对不同领域的内部感知
- 实验证明模型可区分相关领域，微调模型未必最准确
- 适用于开放与闭合生成任务，适合做领域自适应选型
我们研究大型语言模型（LLMs）是否天然捕捉自然语言中的领域特异性。实验通过分析预填充阶段生成的隐藏状态，探测模型对不同领域查询的敏感性。结果揭示出反映模型内部领域识别的潜在轨迹。同时研究了这些领域表征对提示风格和来源变化的鲁棒性。基于此表征实现模型选择：将输入查询的领域轨迹与各模型性能匹配，选出最优适配模型。发现即使在相关领域间，模型也能有效区分；且微调模型并非总是表现最佳。该方法适用于封闭式与开放式生成任务，解释具有通用性。
原文摘要 · Abstract (English)
We study whether Large Language Models (LLMs) inherently capture domain-specific nuances in natural language. Our experiments probe the domain sensitivity of LLMs by examining their ability to distinguish queries from different domains using hidden states generated during the prefill phase. We reveal latent domain-related trajectories that indicate the model's internal recognition of query domains. We also study the robustness of these domain representations to variations in prompt styles and sources. Our approach leverages these representations for model selection, mapping the LLM that best matches the domain trace of the input query (i.e., the model with the highest performance on similar traces). Our findings show that LLMs can differentiate queries for related domains, and that the fine-tuned model is not always the most accurate. Unlike previous work, our interpretations apply to both closed and open-ended generative tasks
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。