Multimodal Functional Maximum Correlation for Emotion Recognition

Zheng, Deyang

doi:10.1109/taffc.2026.3695876

Multimodal Functional Maximum Correlation for Emotion Recognition

2026-07-01

0

PRE

AI

DOI：10.1109/taffc.2026.3695876

Original

Original request for help

Save

Abstract

En 中文

Emotional states manifest as coordinated and heterogeneous physiological responses across central and autonomic systems, posing a fundamental challenge for multimodal representation learning in affective computing. Learning such joint dynamics is further complicated by scarce and subjective affective annotations, motivating the use of self-supervised learning (SSL). However, most existing SSL approaches rely on pairwise alignment objectives, which are insufficient to characterize the joint dependencies among more than two modalities and fail to capture higher-order interactions arising from coordinated brain-autonomic responses. To move beyond this limitation, we propose Multimodal Functional Maximum Correlation (MFMC), a principled SSL framework that maximizes higher-order multimodal dependence via a Dual Total Correlation (DTC) objective. By deriving a tight sandwich bound and optimizing it with a functional maximum correlation analysis (FMCA)-based trace surrogate, MFMC directly captures joint multimodal interactions without resorting to pairwise contrastive losses. Experiments on three public affective computing benchmarks demonstrate that MFMC consistently achieves state-of-the-art or competitive performance under both subject-dependent and subject-independent protocols, highlighting its robustness to inter-subject variability. In particular, MFMC achieves substantial gains on CEAP-360VR, improving subject-dependent accuracy from 78.9% to 86.8% and subject-independent accuracy from 27.5% to 33.1% using the EDA signal alone, and remains highly competitive within 0.8 percentage points of the best-performing method on the most challenging EEG subject-independent split of MAHNOB-HCI.

Keywords:

Electroencephalography

Modeling

Emotion recognition

Labeling

Training

Self-supervised learning

Affective computing

Windows

Correlation

Learning (artificial intelligence)

Multimodal emotion recognition

self-supervised learning

physiological signals

functional maximum correlation analysis

dual total correlation

Multimodal Functional Maximum Correlation for Emotion Recognition

Abstract

Journal

Organization

Cited Papers