Mixed-Modality Clustering via Generative Graph Structure Matching

doi:10.1109/TKDE.2024.3434556

2024-12-01

Xiaxia He

王

王博岳 (Boyue Wang) *

Junbin Gao (Junbin Gao)

Qianqian Wang

Yongli Hu

Baocai Yin

DOI：10.1109/TKDE.2024.3434556

原文链接

原文求助

摘要

En 中文

The goal of mixed-modality clustering, which differs from typical multi-modality/view clustering, is to divide samples derived from various modalities into several clusters. This task has to solve two critical semantic gap problems: i) how to generate the missing modalities without the pairwise-modality data; and ii) how to align the representations of heterogeneous modalities. To tackle the above problems, this paper proposes a novel mixed-modality clustering model, which integrates the missing-modality generation and the heterogeneous modality alignment into a unified framework. During the missing-modality generation process, a bidirectional mapping is established between different modalities, enabling generation of preliminary representations for the missing-modality using information from another modality. Then the intra-modality bipartite graphs are constructed to help generate better missing-modality representations by weighted aggregating existing intra-modality neighbors. In this way, a pairwise-modality representation for each sample can be obtained. In the process of heterogeneous modality alignment, each modality is modelled as a graph to capture the global structure among intra-modality samples and is aligned against the heterogeneous modality representations through the adaptive heterogeneous graph matching module. Experimental results on three public datasets show the effectiveness of the proposed model compared to multiple state-of-the-art multi-modality/view clustering methods.

Keyword:

Correlation

Semantics

Task analysis

Data models

Web sites

Feature extraction

Bipartite graph

Mixed-modality clustering

multi-view clustering

adaptive graph structure learning

heterogeneous graph matching