利用知识几何结构扩展大模型生成多样性边界。
Geometry of Knowledge Allows Extending Diversity Boundaries of Large Language Models
- 基于语义空间的流形结构，通过连续表示条件化生成。
- 小规模锚点生成即可显著提升生成多样性。
- 无需修改模型参数，适合提升创造力相关任务。
我们假设语义空间中的知识沿有结构的流形组织，这种几何结构使空间可探索。通过遍历该空间并用所得连续表示来条件化大型语言模型（LLM）的生成分布，可系统性扩展模型可达的语义范围。我们提出一种无需修改LLM参数的框架，通过少量多样化锚点生成构建条件分布，并借助xRAG风格投影器对LLM生成进行条件化。实验表明，基于流形的条件化显著提升了生成多样性，直接促进模型发散性思维能力，这是创造力的核心特征。
原文摘要 · Abstract (English)
Starting from the hypothesis that knowledge in semantic space is organized along structured manifolds, we argue that this geometric structure renders the space explorable. By traversing it and using the resulting continuous representations to condition an LLM's generation distribution, we can systematically expand the model's reachable semantic range. We introduce a framework that requires no modification of LLM parameters and operationalizes this idea by constructing a conditioning distribution from a small set of diverse anchor generations. This distribution conditions LLM's generation via an xRAG-style projector. Our experiments demonstrate that this manifold-based conditioning substantially increases generative diversity, with direct benefits for enhancing divergent thinking, a core facet of creativity, in language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。