让大模型在可解释与高性能间自由切换,还能自动分配能力。
Localist LLMs with Recruitment Learning
- 用可调参数控制模型表征从局部到分布式过渡,无需重训练。
- 基于信息论动态分配语义模块,初始无需完整领域知识。
- 支持多粒度架构自适应,适合需要透明与性能的监管场景。
我们提出一种新型大语言模型训练框架,可连续调节内部表示,覆盖从局部化(可解释、基于规则)到分布式(泛化性强、高效)的全谱范围。核心创新包括:(1) 局部性旋钮——一个可调参数,可在训练和推理中动态控制定位程度,无需重训练;(2) 基于信息论的招募机制,按需自适应分配语义块,避免初始化时需完全领域知识;(3) 分层招募框架,将容量分配扩展至整个专用LLM,实现多粒度架构适配。方法通过注意力机制的组稀疏惩罚、信息论锚点设计、动态规则注入及基于惩罚似然的明确单元招募准则实现。我们提供严格的数学证明,确立注意力在稳态点集中于语义相关块的显式阈值条件,给出注意力熵与指针保真度的精确边界。分层招募机制在块级(细粒度,模型内)与模型级(粗粒度,跨领域)均提供收敛保证,确保系统发现平衡模型复杂度与数据编码效率的语义划分。该框架使从业者能持续在可解释与高性能模式间切换,并在多粒度上适应架构容量,适用于需兼顾透明性与能力的监管领域应用。
原文摘要 · Abstract (English)
We present a novel framework for training large language models with continuously adjustable internal representations that span the full spectrum from localist (interpretable, rule-based) to distributed (generalizable, efficient) encodings. The key innovations are (1) a locality dial, a tunable parameter that dynamically controls the degree of localization during both training and inference without requiring model retraining, (2) an information-theoretic recruitment mechanism that adaptively allocates semantic blocks as needed, eliminating the requirement for complete domain knowledge at initialization, and (3) a hierarchical recruitment framework that extends capacity allocation to entire specialized LLMs, enabling multi-granularity architectural adaptation. This is achieved through group sparsity penalties on attention mechanisms, information-theoretic anchor design, dynamic rule injection, and principled recruitment criteria based on penalized likelihood with explicit units. We provide rigorous mathematical results establishing explicit threshold conditions under which attention provably concentrates on semantically relevant blocks at stationary points, with exact bounds on attention entropy and pointer fidelity. The hierarchical recruitment mechanism provides convergence guarantees at both the block level (fine-grained, within-LLM) and the LLM level (coarse-grained, cross-domain), ensuring the system discovers semantic partitions that balance model complexity against data encoding efficiency. This framework enables practitioners to continuously interpolate between interpretable and high-performance modes while adapting architectural capacity at multiple granularities, supporting applications in regulated domains requiring both transparency and capability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。