AI Learns to Pick the Best Clustering Algorithm Before Any Data Is Labeled

Every practicing data scientist knows the quiet agony of the unlabeled spreadsheet. Supervised classification, the workhorse of modern machine learning, depends on human annotators who painstakingly assign categories to each row of a table, whether those rows describe patients, customers, sensor readings, or financial transactions. Annotation is slow, expensive, and often the single largest bottleneck in a machine learning pipeline. A tempting alternative is to let clustering algorithms generate so-called pseudo-labels automatically: group similar rows together, treat each group as a class, and train a classifier on the result without paying for a single human label. The catch, as a new study in the International Journal of Data Science and Analytics demonstrates, is that the choice of clustering algorithm matters enormously, and no single method wins everywhere.

Researchers Karim Hallal, Adam Dandan, Sireen Hammoud and Seifedine Kadry of the Lebanese American University have now tackled this selection problem head-on with a framework they describe as meta-learning for pseudo-label utility prediction. Their central insight is deceptively simple: rather than running every candidate clustering algorithm on every new dataset, one can learn from past experience which algorithm is likely to serve a given dataset best. The team introduces a metric called Label Substitution Efficiency, or LSE, which quantifies how well pseudo-labels can stand in for human annotation. LSE is computed by training two classifiers on the same dataset, one on pseudo-labels produced by a clustering algorithm and one on the ground-truth labels, and then comparing their balanced accuracy. If the pseudo-label-trained classifier nearly matches the ground-truth-trained one, the clustering algorithm has done its job of substituting for the annotator.

The technical machinery behind the study is worth unpacking. The authors benchmark six pseudo-label generators across 94 datasets drawn from OpenML, the open repository of machine learning datasets that has become a standard testbed for meta-learning research. For each dataset and each clustering algorithm, they compute a six-dimensional LSE vector, one entry per algorithm, capturing the full profile of how each candidate performs as a label substitute. This vector becomes the prediction target for the meta-learners. The idea of using downstream classification performance as the yardstick for clustering quality is itself a departure from tradition. Classical clustering validation indices, such as silhouette scores or Davies-Bouldin measures, judge clusters by their internal geometry. LSE instead asks a more practical question: do these clusters, converted into labels, actually teach a classifier something useful?

With the LSE tables in hand, the researchers trained two complementary families of meta-learners. The first is a classification meta-learner that treats algorithm selection as a discrete recommendation problem: given a description of a new dataset, it directly outputs the clustering method it predicts will yield the highest LSE. The second is a regression meta-learner that predicts the entire six-dimensional LSE vector, offering far more flexibility. Because it estimates the utility of every candidate rather than committing to a single winner, the regression model supports confidence thresholding, ranking and shortlisting strategies. A practitioner could, for example, ask for the top three algorithms and run only those, or reject a recommendation outright if the predicted utility falls below a threshold. This graded output acknowledges a truth that discrete recommenders often gloss over: sometimes no clustering algorithm will substitute well for labels, and it is better to know that before investing compute.

A crucial design question in any meta-learning system is how to describe a dataset to the learner. These descriptions, called meta-features, traditionally include simple statistical summaries such as the number of instances, the number of attributes, class entropy proxies, correlation statistics and measures of skewness or dimensionality. The study evaluates five different meta-feature representations, including a novel one the authors call Option C2, a multi-scale dictionary learning representation. Dictionary learning, a technique rooted in sparse coding, represents each dataset as a sparse combination of learned basis elements across multiple scales, potentially capturing structural signatures that hand-crafted statistics miss. The approach draws on earlier work on coupled dictionary learning for unsupervised feature selection and on online matrix factorization methods developed by Mairal and colleagues.

The headline result is striking. The best meta-classifier, a logistic regression model operating on the C2 dictionary-learning features, achieves 43.6 percent top-1 accuracy under leave-one-out validation, meaning it picks the single best clustering algorithm for a previously unseen dataset nearly half the time. That figure may sound modest until it is compared against the baseline: the classic de Souto ranking approach from 2008, a foundational meta-learning method for clustering algorithm selection, manages only 22.3 percent on the same task. In other words, the new framework nearly doubles the performance of the established default. In a domain where the best algorithm varies substantially across datasets and where running all candidates defeats the purpose of annotation-free learning, that improvement translates into real savings of time and computation.

Statistical rigor underpins the claims. Paired significance testing shows that the meta-learner significantly outperforms a weak heuristic baseline with a p-value of 0.003, a result that would clear conventional thresholds for statistical significance with room to spare. Perhaps more interesting is what the tests did not find: no single meta-feature representation significantly outperformed another, with all pairwise comparisons yielding p-values above 0.08. The authors interpret this as evidence that the framework’s benefit is robust to the specific feature-engineering choice rather than dependent on any one representation. For practitioners, that is good news. It suggests the framework does not hinge on a fragile, bespoke encoding of datasets; the meta-learning signal is strong enough to survive whatever reasonable description of the data one feeds it.

The study situates itself within a rapidly growing literature on automated algorithm selection. Earlier systems such as cSmartML combined meta-learning with hyperparameter tuning for clustering, while more recent efforts like CLAMS have pursued zero-shot model selection, and deep learning approaches such as ClustRecNet have attempted end-to-end recommendation of clustering pipelines. Work on Dataset2Vec has even explored learning meta-features directly from raw data rather than computing them by hand. What distinguishes the new framework is its target: instead of predicting abstract clustering quality, it optimizes for downstream classification efficiency, tying the recommendation directly to the end goal of building a working classifier from pseudo-labels. The authors also employ SHAP, the game-theoretic attribution method introduced by Lundberg and Lee, to interpret which meta-features drive the recommendations, adding a layer of explainability to what could otherwise be an opaque selection process.

The practical implications extend well beyond the benchmark. Pseudo-labeling has become a staple technique in domains where labels are scarce, from remote sensing applications that delineate snow cover in satellite imagery to self-training pipelines for tabular data in medicine and finance. In each of these settings, someone must currently choose a clustering algorithm, often by trial and error, and each trial consumes computational resources that scale with dataset size. A meta-learner that recommends the right algorithm from dataset properties alone, before any clustering is run, collapses that search to a single prediction. The regression variant’s shortlisting capability offers a middle path for risk-averse users: run only the top-ranked candidates and stop early if one achieves a predicted utility above a confidence threshold.

The authors have made their work reproducible and accessible. All 94 OpenML dataset identifiers and the computed LSE tables are publicly available, along with the complete codebase on GitHub, inviting the community to extend the benchmark, test additional meta-feature representations and plug in new clustering algorithms. As annotation costs continue to rise with the growing scale of tabular data in industry and science, frameworks like this one point toward a future in which machines not only learn from data but also learn which learning strategy to use, quietly and automatically, before a single human label is ever requested.

Subject of Research: Meta-learning for unsupervised clustering algorithm selection via pseudo-label utility prediction

Article Title: Meta-learning for pseudo-label utility prediction: unsupervised algorithm selection via downstream classification efficiency

Article References: Hallal, K., Dandan, A., Hammoud, S., & Kadry, S. (2026). Meta-learning for pseudo-label utility prediction: unsupervised algorithm selection via downstream classification efficiency. International Journal of Data Science and Analytics, 22(1), Article 331. https://doi.org/10.1007/s41060-026-01296-2

Image Credits: AI Generated

DOI: 10.1007/s41060-026-01296-2

Keywords: meta-learning, algorithm selection, pseudo-labeling, clustering, Label Substitution Efficiency, OpenML, dictionary learning, tabular data, unsupervised learning, classification, SHAP, automated machine learning