提出几何约束的高效微调方法,提升社科概念检索精度。
OrthoGeoLoRA: Geometric Parameter-Efficient Fine-Tuning for Structured Social Science Concept Retrieval on theWeb
- 通过正交约束重构低秩更新,解决标准LoRA的几何缺陷
- 在相同参数预算下,排名指标显著优于标准LoRA及其他变体
- 适合资源受限机构在数字社科系统中高效适配大模型
大型语言模型和文本编码器正日益驱动社会科学研究领域的网络信息体系,包括数字图书馆、数据目录和研究者、政策制定者及民间组织使用的搜索界面。全量微调计算与能耗成本高,对小型机构和非营利组织构成障碍。参数高效微调(PEFT)特别是低秩适应(LoRA)通过仅更新少量参数降低开销。我们发现标准LoRA更新ΔW = BAᵀ存在几何缺陷:规范自由度、尺度模糊性以及秩坍缩倾向。为此提出OrthoGeoLoRA,通过约束低秩因子为正交(施蒂费尔流形),实现类似SVD的形式ΔW = BΣAᵀ。几何重参数化在保持与Adam等优化器兼容的同时,实现该约束。我们还构建了针对欧洲语言社会科学术语表(ELSST)的层级概念检索基准。多语言句向量编码器实验表明,OrthoGeoLoRA在相同低秩预算下,在排名指标上超越标准LoRA及多种强基线方法,为资源受限场景下的基础模型适配提供更高效路径。
原文摘要 · Abstract (English)
Large language models and text encoders increasingly power web-based information systems in the social sciences, including digital libraries, data catalogues, and search interfaces used by researchers, policymakers, and civil society. Full fine-tuning is often computationally and energy intensive, which can be prohibitive for smaller institutions and non-profit organizations in the Web4Good ecosystem. Parameter-Efficient Fine-Tuning (PEFT), especially Low-Rank Adaptation (LoRA), reduces this cost by updating only a small number of parameters. We show that the standard LoRA update $ΔW = BA^\top$ has geometric drawbacks: gauge freedom, scale ambiguity, and a tendency toward rank collapse. We introduce OrthoGeoLoRA, which enforces an SVD-like form $ΔW = BΣA^\top$ by constraining the low-rank factors to be orthogonal (Stiefel manifold). A geometric reparameterization implements this constraint while remaining compatible with standard optimizers such as Adam and existing fine-tuning pipelines. We also propose a benchmark for hierarchical concept retrieval over the European Language Social Science Thesaurus (ELSST), widely used to organize social science resources in digital repositories. Experiments with a multilingual sentence encoder show that OrthoGeoLoRA outperforms standard LoRA and several strong PEFT variants on ranking metrics under the same low-rank budget, offering a more compute- and parameter-efficient path to adapt foundation models in resource-constrained settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。