用规则引导多模态聚类,让模型理解领域知识而非只看外表相似。
Beyond Visual Similarity: Rule-Guided Multimodal Clustering with explicit domain rules
- 将领域规则嵌入变分自编码器,作为学习信号直接指导聚类
- 在飞机和汽车数据集上,聚类结果更符合实际分类(如区分无人机与隐身战机)
- 适合需要专业知识的复杂场景,如军工、医疗等高要求领域
传统聚类方法仅依赖输入数据的相似性,难以捕捉结构或语义约束。本文提出领域感知规则触发变分自编码器(DARTVAE),将领域特定约束直接融入表示学习过程。DARTVAE通过将显式规则、语义表示和数据驱动特征统一编码至潜在空间,并在损失函数中引入规则一致性与违反惩罚,强制遵守约束。规则由大语言模型生成并构建成知识图谱,作为学习信号。实验表明,在飞机和汽车数据集上,该方法能生成更具操作意义且可解释的聚类结果,例如分离无人机、合并隐身战机、区分SUV与轿车,同时提升传统聚类指标。但该框架存在挑战:大模型生成规则可能幻觉或冲突,规则过多易导致过拟合,复杂领域扩展时计算和一致性难度上升。结合规则编码与学习表示,DARTVAE在复杂知识密集型场景下优于纯数据驱动模型。
原文摘要 · Abstract (English)
Traditional clustering techniques often rely solely on similarity in the input data, limiting their ability to capture structural or semantic constraints that are critical in many domains. We introduce the Domain Aware Rule Triggered Variational Autoencoder (DARTVAE), a rule guided multimodal clustering framework that incorporates domain specific constraints directly into the representation learning process. DARTVAE extends the VAE architecture by embedding explicit rules, semantic representations, and data driven features into a unified latent space, while enforcing constraint compliance through rule consistency and violation penalties in the loss function. Unlike conventional clustering methods that rely only on visual similarity or apply rules as post hoc filters, DARTVAE treats rules as first class learning signals. The rules are generated by LLMs, structured into knowledge graphs, and enforced through a loss function combining reconstruction, KL divergence, consistency, and violation penalties. Experiments on aircraft and automotive datasets demonstrate that rule guided clustering produces more operationally meaningful and interpretable clusters for example, isolating UAVs, unifying stealth aircraft, or separating SUVs from sedans while improving traditional clustering metrics. However, the framework faces challenges: LLM generated rules may hallucinate or conflict, excessive rules risk overfitting, and scaling to complex domains increases computational and consistency difficulties. By combining rule encodings with learned representations, DARTVAE achieves more meaningful and consistent clustering outcomes than purely data driven models, highlighting the utility of constraint guided multimodal clustering for complex, knowledge intensive settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。