.png){kind=icon}
返回
Hypercube Pooling for Visual Semantic Embedding
.png){kind=icon}
OA
AI
DOI：10.1145/3689637.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Visual Semantic Embedding (VSE) is a primary model for cross-modal retrieval, wherein the global feature aggregator is a crucial component of the VSE model. In recent research, the General Pooling Operator (GPO) aggregator, which weighs the features reconstructed from the local feature set to aggregate, facilitates the related models to achieve good retrieval performance. However, the reason for the effectiveness remains to be explored. To enhance the rationality of aggregator designs, we analyze the reason from the perspective of feature space. Indeed, for each data, the local feature set forms a hypercube containing abundant data information, and the feature learned by GPO measures the hypercube, thereby representing the data. The geometric structure of the hypercube implies that the set containing all points within the hypercube is a convex set, so the feature learned by weighted aggregation is an interior point of the hypercube. However, using the interior point to measure the hypercube leads to some problems in feature representation and model optimization, as well as the reduction of retrieval efficiency caused by weight computation. For example, the related pair's features may be far, while the unrelated ones may be close. To measure the hypercube more clearly and alleviate the problems mentioned above, we propose Hypercube Pooling (HCP) aggregator. Specifically, HCP concatenates the Max and Min Pooling features as the global features. This aggregation method has multiple advantages, e.g., the learned global feature represents all hyperplanes of the hypercube that contain critical information and hypercube geometric structure. Moreover, HCP adds normalizationbefore-concatenation and reduces the usual setting of margin in the loss function by half to avoid gradient loss caused by the difference in the feature value and dimensionality doubling. The experimental results on the Flickr30K and MSCOCO datasets show that the HCP model has excellent performance with high efficiency, confirming the correctness of the spatial analysis and the effectiveness of the HCP aggregator.
Keyword:
Cross-Modal Retrieval
Visual Semantic Embedding
Global Pooling
Hypercube
期刊
IF:
6
论文数:
2.0K
被引数:
5.4K
机构
暂无机构信息
引用论文
.png){kind=icon}
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations视觉基因组: 使用众包密集图像注释连接语言和视觉
OAAI
OAAI分享收藏分享收藏MKVSE: Multimodal Knowledge Enhanced Visual-semantic Embedding for Image-text RetrievalMKVSE: 用于图像文本检索的多模态知识增强的视觉语义嵌入OAAI
OAAI分享收藏
查看更多内容