通过分层嵌入对齐提升大模型检索准确性，减少幻觉。
HEAL: Hierarchical Embedding Alignment Loss for Improved Retrieval and Representation Learning
- 用分层模糊聚类与矩阵分解设计对比学习损失
- 在医疗、材料等多领域显著提升检索相关性
- 适合需要精准知识检索的垂直场景应用
检索增强生成（RAG）通过引入外部文档检索，使大语言模型（LLM）能够获取领域特定或最新知识，从而提升性能。RAG的效果依赖于检索文档的相关性，而这一相关性受模型嵌入与领域内容语义对齐程度的影响。尽管全量微调可实现领域对齐，但其计算成本高且需大量数据。本文提出分层嵌入对齐损失（HEAL），在对比学习框架中结合分层模糊聚类与矩阵分解，高效对齐LLM嵌入与领域内容。HEAL计算各层级/深度的对比损失，并引入层次惩罚项，以对齐标签体系中的潜在关系。该方法提升了检索相关性与文档分类效果，有效降低大模型输出的幻觉。我们在医疗、材料科学、网络安全和应用数学等多个领域进行基准测试与评估，验证了HEAL的有效性。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by integrating external document retrieval to provide domain-specific or up-to-date knowledge. The effectiveness of RAG depends on the relevance of retrieved documents, which is influenced by the semantic alignment of embeddings with the domain's specialized content. Although full fine-tuning can align language models to specific domains, it is computationally intensive and demands substantial data. This paper introduces Hierarchical Embedding Alignment Loss (HEAL), a novel method that leverages hierarchical fuzzy clustering with matrix factorization within contrastive learning to efficiently align LLM embeddings with domain-specific content. HEAL computes level/depth-wise contrastive losses and incorporates hierarchical penalties to align embeddings with the underlying relationships in label hierarchies. This approach enhances retrieval relevance and document classification, effectively reducing hallucinations in LLM outputs. In our experiments, we benchmark and evaluate HEAL across diverse domains, including Healthcare, Material Science, Cyber-security, and Applied Maths.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。