Relation-Aware Distribution Representation Network for Person Clustering With Multiple Modalities

Kaijian Liu

doi:10.1109/TMM.2023.3304454

Relation-Aware Distribution Representation Network for Person Clustering With Multiple Modalities

2024-01-01

0

OA

AI

DOI：10.1109/TMM.2023.3304454

Original

Original request for help

Save

Abstract

En 中文

Person clustering with multi-modal clues, including faces, bodies, and voices, is critical for various tasks, such as movie parsing and identity-based movie editing. Related methods such as multi-view clustering mainly project multi-modal features into a joint feature space. However, multi-modal clue features are usually rather weakly correlated due to the semantic gap from the modality-specific uniqueness. As a result, these methods are not suitable for person clustering. In this article, we propose a Relation-Aware Distribution representation Network (RAD-Net) to generate a distribution representation for multi-modal clues. The distribution representation of a clue is a vector consisting of the relation between this clue and all other clues from all modalities, thus being modality agnostic and good for person clustering. Accordingly, we introduce a graph-based method to construct distribution representation and employ a cyclic update policy to refine distribution representation progressively. Our method achieves substantial improvements of +6% and +8.2% in F-score on the Video Person-Clustering Dataset (VPCD) and VoxCeleb2 multi-view clustering dataset, respectively.

Keywords:

Faces

Feature extraction

Streaming media

Task analysis

Measurement

Semantics

Motion pictures

Person clustering

Multi-modality clues

Distribution learning

Multi-modal representations

Relation-Aware Distribution Representation Network for Person Clustering With Multiple Modalities

Abstract

Journal

Organization