Caption-Aided Product Detection via Collaborative Pseudo-Label Harmonization

doi:10.1109/TMM.2022.3222653

Caption-Aided Product Detection via Collaborative Pseudo-Label Harmonization

2023-01-01

2

PRE

AI

DOI：10.1109/TMM.2022.3222653

Original

Original request for help

Save

Abstract

En 中文

Product detection, which aims to localize products of interest in the advertising images, helps advance many potential E-commerce applications like product retrieval and recommendation. However, labeling a massive number of fine-grained product categories and accurate product boxes is costly and especially not practical since products are ever-changing on E-commerce websites. In this work, we step forward to train a fine-grained product detector solely supervised by the advertising captions, which are naturally available but often severely flawed and noisy. To reformulate the weakly supervised detection research into a real-world setting, we introduce a large-scale benchmark, named CapProduct, where more than 80,000 product image-caption pairs are collected from E-commerce websites. The fine-grained nature of products and noisy captions in CapProduct make it intractable to excavate valid category labels to train a weakly supervised object detector. To tackle this challenge, we propose a Collaborative Pseudo-Label Harmonization (CoPLH) framework that harmonizes self-mined pseudo labels via modeling the global co-occurrence relationships of products. We construct a collaborative co-occurrence graph based on all training samples to improve the reliability of caption-predicted pseudo-labels as well as benefit the self-training procedure in a weakly supervised setting. Extensive experiments on the CapProduct dataset demonstrate the effectiveness and the superiority of the proposed CoPLH over the state-of-the-art baselines.

Keywords:

Product detection

pseudo-label

positive mining

Caption-Aided Product Detection via Collaborative Pseudo-Label Harmonization

Abstract

Journal

Organization