.png){kind=icon}
Return
(Un)likelihood Training for Interpretable Embedding
.png){kind=icon}
OA
AI
DOI：10.1145/3632752.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Cross-modal representation learning has become a new normal for bridging the semantic gap between text and visual data. Learning modality agnostic representations in a continuous latent space, however, is often treated as a black-box data-driven training process. It is well known that the effectiveness of representation learning depends heavily on the quality and scale of training data. For video representation learning, having a complete set of labels that annotate the full spectrum of video content for training is highly difficult, if not impossible. These issues, black-box training and dataset bias, make representation learning practically challenging to be deployed for video understanding due to unexplainable and unpredictable results. In this article, we propose two novel training objectives, likelihood and unlikelihood functions, to unroll the semantics behind embeddings while addressing the label sparsity problem in training. The likelihood training aims to interpret semantics of embeddings beyond training labels, while the unlikelihood training leverages prior knowledge for regularization to ensure semantically coherent interpretation. With both training objectives, a new encoder-decoder network, which learns interpretable cross-modal representation, is proposed for ad-hoc video search. Extensive experiments on TRECVid and MSR-VTT datasets show that the proposed network outperforms several state-of-the-art retrieval models with a statistically significant performance margin.
Keywords:
Explainable embedding
cross-modal representation learning
ad-hoc video search
Journal
IF:
9.1
Papers:
1.2K
Citations:
4.7K
Organization
Cited Papers
Studies on the control of diploid-like meiosis in polyploid taxa of Chrysanthemum. I. Hexaploid Ch. japonense Nakai..png)
CYTOLOGIA
OAAI
CYTOLOGIAIF0
OAAIShareSaveShareSaveShareSaveShareSaveSynthesis of n-type semiconducting diamond film using diphosphorus pentaoxide as the doping source
PREAI
ShareSaveThe Outcome of Phagocytic Cell Division with Infectious Cargo Depends on Single Phagosome FormationPLoS ONEOAAI
PLoS ONEIF0
OAAIShareSaveShareSave
View more