Meta-learning for pseudo-label utility prediction: unsupervised algorithm selection via downstream classification efficiency

doi:10.1007/s41060-026-01296-2

Meta-learning for pseudo-label utility prediction: unsupervised algorithm selection via downstream classification efficiency

2026-10-07

0

PRE

AI

DOI：10.1007/s41060-026-01296-2

Original

Original request for help

Save

Abstract

En 中文

Annotating tabular data for supervised classification is costly and time-consuming. Clustering-based pseudo-labeling offers a zero annotation alternative, yet choosing the right clustering algorithm is non-trivial: the best method varies substantially across datasets, and exhaustively running all candidates defeats the purpose of annotation-free learning. We introduce Label Substitution Efficiency (LSE), a novel metric that measures how well pseudo-labels substitute for human annotation by comparing the balanced accuracy of a classifier trained on pseudo-labels against one trained on ground-truth labels. Using LSE as both a discrete recommendation target and a continuous regression target, we train two complementary families of meta-learners that recommend the best clustering algorithm from dataset properties alone, before any clustering is run: a classification meta-learner that emits the recommended method directly, and a regression meta-learner that predicts the full six-dimensional LSE vector and supports confidence thresholding, ranking, and shortlisting strategies. We benchmark six pseudo-label generators across 94 OpenML datasets and evaluate five meta-feature representations, including a novel multi-scale dictionary learning representation (Option C2). Our best meta-classifier, logistic regression on C2 features, achieves 43.6% top-1 leave-one-out accuracy, nearly doubling the de Souto default-ranking baseline (22.3%). Paired significance testing shows the meta-learner significantly outperforms a weak heuristic baseline ( $$p=0.003$$ ), though no single meta-feature representation significantly outperforms another ( $$p > 0.08$$ for all pairwise comparisons), indicating the framework’s benefit is robust to the specific feature-engineering choice rather than dependent on any one representation.

Keywords:

Meta-learning

Algorithm selection

Pseudo-labeling

Clustering

Dictionary learning

SHAP

Tabular data.

Meta-learning for pseudo-label utility prediction: unsupervised algorithm selection via downstream classification efficiency

Abstract

Journal

Organization

Cited Papers