Wasserstein task embedding for measuring task similarities

Xinran Liu

doi:10.1016/j.neunet.2024.106796

Wasserstein task embedding for measuring task similarities

2025-01-01

0

OA

AI

DOI：10.1016/j.neunet.2024.106796

Original

Save

View PDF

Abstract

En 中文

Measuring similarities between different tasks is critical in abroad spectrum of machine learning problems, including transfer, multi-task, continual, and meta-learning. Most current approaches to measuring task similarities are architecture-dependent: (1) relying on pre-trained models, or (2) training networks on tasks and using forward transfer as a proxy for task similarity. In this paper, we leverage the optimal transport theory and define a novel task embedding for supervised classification that is model-agnostic, training-free, and capable of handling (partially) disjoint label sets. In short, given a dataset with ground-truth labels, we perform a label embedding through multi-dimensional scaling and concatenate dataset samples with their corresponding label embeddings. Then, we define the distance between two datasets as the 2-Wasserstein distance between their updated samples. Lastly, we leverage the 2-Wasserstein embedding framework to embed tasks into a vector space in which the Euclidean distance between the embedded points approximates the proposed 2-Wasserstein distance between tasks. We show that the proposed embedding leads to a significantly faster comparison of tasks compared to related approaches like the Optimal Transport Dataset Distance (OTDD). Furthermore, we demonstrate the effectiveness of our embedding through various numerical experiments and show statistically significant correlations between our proposed distance and the forward and backward transfer among tasks on a wide variety of image recognition datasets.

Keywords:

Task embedding

Dataset similarity

Optimal transport

Continual learning

Transfer learning

Wasserstein task embedding for measuring task similarities

Abstract

AI Summary

Journal

Organization