用随机句子增强文本表示，提升大模型对群体偏见的鲁棒性
DoubleCCA: Improving Foundation Model Group Robustness with Random Sentence Embeddings
- 通过随机句扩充提示，生成多样化文本嵌入
- 两次使用CCA对齐并重构表示，增强模型泛化能力
- 方法简单易集成，适合改进现有大模型的公平性
本文提出一种新方法DoubleCCA，用于提升基础模型对群体偏差的鲁棒性。首先，生成包含随机词或字符序列的多样化随机句子，以扩充原始提示；其次，利用额外的句子嵌入模型为这些随机句子生成不同文本嵌入；随后，通过两次应用典型相关分析（CCA）对齐表示，并将结果重构回原始表示空间。在多个任务和数据集上的实验表明，该方法在性能与鲁棒性上均优于现有方法。其结构简单，易于集成到现有模型中，是一种实用的提升基础模型抗群体偏差能力的解决方案。
原文摘要 · Abstract (English)
This paper presents a novel method to improve the robustness of foundation models to group-based biases. We propose a simple yet effective method, called DoubleCCA, that leverages random sentences and Canonical Correlation Analysis (CCA) to enrich the text embeddings of the foundation model. First, we generate various random sentences that augment the original prompts, which extends the original prompts with random words or character sequences. Second, we use an additional sentence embedding model to generate different text embeddings with respect to these random sentences. We then use CCA double twice to align the representations and reconstruct them back to the original representation space. We demonstrate the effectiveness of our method on a variety of tasks and datasets, showing that it outperforms existing methods in terms of both performance and robustness. Our method is simple to implement and can be easily integrated into existing models, making it a practical solution for improving the robustness of foundation models to group-based biases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。