用大模型提炼可信检索器,省去实时调用大模型
MACA: A Framework for Distilling Trustworthy LLMs into Efficient Retrievers
- 将校准过的元数据感知大模型转化为轻量检索器
- 在银行问答数据集上准确率提升至0.48,比基线高25个百分点
- 适合需要高效、低成本且支持生成式检索的金融场景
现代企业检索系统需处理如'海外交易手续费退款'、'近期支票状态'等简短模糊查询。此时语义细微差别和元数据至关重要,但每查询调用大语言模型重排序和人工标注成本高昂。本文提出元数据感知跨模型对齐框架MACA,将校准的元数据感知大模型重排序器蒸馏为紧凑的学生检索器,避免在线调用大模型。通过元数据感知提示验证教师模型可信性,检查其在排列变换和改写下的一致性与鲁棒性,并提供列表级得分、困难负样本及校准的相关性阈值。学生模型采用MACA的MetaFusion目标函数,结合元数据条件排名损失与跨模型边界损失,学习将正确答案推至语义相似但主题、子主题或实体不匹配的候选项之上。在自研消费银行业务问答语料库和BankFAQs数据集上,MACA教师模型在Accuracy@1上分别较MAFA基线提升5个百分点和3个百分点。学生模型显著优于预训练编码器:例如在自研语料库上,MiniLM的Accuracy@1从0.23提升至0.48,同时保持推理无需大模型调用,支持检索增强生成。
原文摘要 · Abstract (English)
Modern enterprise retrieval systems must handle short, underspecified queries such as ``foreign transaction fee refund'' and ``recent check status''. In these cases, semantic nuance and metadata matter but per-query large language model (LLM) re-ranking and manual labeling are costly. We present Metadata-Aware Cross-Model Alignment (MACA), which distills a calibrated metadata aware LLM re-ranker into a compact student retriever, avoiding online LLM calls. A metadata-aware prompt verifies the teacher's trustworthiness by checking consistency under permutations and robustness to paraphrases, then supplies listwise scores, hard negatives, and calibrated relevance margins. The student trains with MACA's MetaFusion objective, which combines a metadata conditioned ranking loss with a cross model margin loss so it learns to push the correct answer above semantically similar candidates with mismatched topic, sub-topic, or entity. On a proprietary consumer banking FAQ corpus and BankFAQs, the MACA teacher surpasses a MAFA baseline at Accuracy@1 by five points on the proprietary set and three points on BankFAQs. MACA students substantially outperform pretrained encoders; e.g., on the proprietary corpus MiniLM Accuracy@1 improves from 0.23 to 0.48, while keeping inference free of LLM calls and supporting retrieval-augmented generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。