.png){kind=icon}
Return
Bayesian mixture variational autoencoders for multi-modal learning
.png){kind=icon}
OA
AI
DOI：10.1007/s10994-022-06272-y.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
This paper provides an in-depth analysis on how to effectively acquire and generalize cross-modal knowledge for multi-modal learning. Mixture-of-Expert (MoE) and Product-of-Expert (PoE) are two popular directions in generalizing multi-modal information. Existing works based on MoE or PoE have shown notable improvement on data generation, while new challenges such as high training cost, overconfident experts, and encoding modal-specific features also emerge. In this work, we propose Bayesian mixture variational autoencoder (BMVAE) which learns to select or combine experts via Bayesian inference. We show that the proposed idea can naturally encourage models to learn modal-specific knowledge and avoid overconfident experts. Also, we show that the idea is compatible with both MoE and PoE frameworks. When being a MoE model, BMVAE can be optimized by a tight lower bound and is efficient to train. The PoE BMVAE has the same advantages and a theoretical connection to existing works. In the experiments, we show that BMVAE achieves state-of-the-art performance.
Keywords:
Multi-modal learning
Mixture-of-Gaussian
Product-of-Gaussian
Deep generative model
Disentangled representation
Journal
IF:
2.9
Papers:
2.7K
Citations:
3.4W
Organization
Cited Papers
.png){kind=icon}
ShareSaveShareSaveShareSave no more
no more