用分块数据与低秩微调，高效移除大模型敏感内容而不伤及通用知识。
AILS-NTUA at SemEval-2025 Task 4: Parameter-Efficient Unlearning for Large Language Models using Data Chunking
- 通过数据分块和循环采样保留数据，配合低秩微调实现参数高效遗忘。
- 在基准测试中表现最优，遗忘效果显著优于现有方法。
- 适用于需要安全可控的模型更新场景，如隐私保护与内容净化。
大语言模型去除敏感内容任务旨在移除特定训练数据点，同时尽可能保留模型的通用知识。本文提出一种参数高效的梯度式遗忘方法，结合低秩适配（LoRA）与层聚焦微调。为进一步提升遗忘效果，采用数据分块策略：将需遗忘的数据拆分为互不重叠的片段，并以预设比例与循环采样的保留数据混合。该任务无关方法在排行榜上排名第一，显著优于基线与竞争系统，实现了出色的遗忘-保留平衡。
原文摘要 · Abstract (English)
The Unlearning Sensitive Content from Large Language Models task aims to remove targeted datapoints from trained models while minimally affecting their general knowledge. In our work, we leverage parameter-efficient, gradient-based unlearning using low-rank (LoRA) adaptation and layer-focused fine-tuning. To further enhance unlearning effectiveness, we employ data chunking, splitting forget data into disjoint partitions and merging them with cyclically sampled retain samples at a pre-defined ratio. Our task-agnostic method achieves an outstanding forget-retain balance, ranking first on leaderboards and significantly outperforming baselines and competing systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。