When machine learning models are asked to predict how satisfied users will be with a finished software project, the training data matters more than its polish. A study published in Knowledge and Information Systems reports that models trained on larger datasets containing low-quality records usually outperformed models trained on smaller, carefully vetted ones. The work, conducted by Łukasz Radliński using the repository of the International Software Benchmarking Standards Group (ISBSG), suggests that for this prediction task, expanding the training data can improve accuracy even when the added cases carry lower data-quality ratings. The finding challenges a long-standing guideline in the benchmarking community, which advises researchers to discard poorly rated projects before modelling begins.
User satisfaction is one component of what standards bodies call quality in use, which the ISO and IEC define as the degree to which a product can be used by specific users to meet their needs with effectiveness, efficiency, satisfaction and freedom from risk. Because satisfaction is a subjective construct, it cannot be measured as directly as, say, the size of a program; it is typically captured through surveys summarising the views of users and other stakeholders. Collecting such data when a project is released allows organisations to act on dissatisfaction in ongoing work or to anticipate satisfaction levels in future projects. The ISBSG dataset, which contains records of thousands of software projects submitted voluntarily, is one of the few publicly available sources that includes attributes reflecting user satisfaction, which is why the study centred on it.
That choice came with a complication. ISBSG guidelines recommend using only projects rated ‘A’ or ‘B’ on two quality attributes, the Data Quality Rating and the UFP Rating, where ‘A’ denotes nothing suspicious in the data and ‘D’ reflects little data credibility. Most prior studies followed this advice, sometimes restricting the data even further to rating ‘A’ alone. Yet the ISBSG itself has acknowledged that C- and D-rated projects may still provide valuable data. Filtering by rating shrinks the pool of usable cases, and for a target such as user satisfaction, which only a small fraction of projects report, the resulting training set can become too small for machine learning to work with at all. The study set out to test whether that situation is truly irrecoverable.
The experimental design was deliberately broad. The researcher constructed 60 data variants, each defined by three thresholds: how restrictive the Data Quality Rating filter was, how restrictive the UFP Rating filter was, and a threshold for the fraction of valid values below which predictor attributes were discarded. These variants were crossed with twelve predictive models, including two naive baselines and ten machine learning algorithms, yielding 720 model-and-data combinations. Missing values, which pervade the ISBSG data, were filled with simple imputation, using the mean for numeric attributes and the mode for non-numeric ones. The use of principal component analysis was treated as a binary hyperparameter, so that it was applied only when cross-validation showed it helped for a given combination of data variant, model and settings.
To keep the conclusions reliable, the study employed repeated nested cross-validation: an inner fivefold cross-validation loop tuned hyperparameters, while an outer fivefold loop, repeated twenty times with different random splits, produced the final evaluations. Performance was aggregated with three measures: standardised accuracy, which expresses how much better predictions were than random guessing; a win–tie–loss statistic based on pairwise Wilcoxon signed-rank tests of absolute-error distributions; and the mean matched-pairs rank biserial correlation, an effect-size measure. The Scott–Knott effect size difference test grouped treatments into statistically distinct performance tiers, and Kendall’s coefficient of concordance assessed whether rankings held steady across data splits and across grouping dimensions.
The results were strikingly consistent. For nearly all non-baseline models, at least one data variant from the permissive group that retained all cases regardless of rating reached the best rank on every evaluation measure. No variant based on the strictest ‘A’-only ratings ever achieved the best rank for any non-baseline model. Among the algorithms, support vector machines and extreme gradient boosting trees delivered the most accurate predictions across all data variants, and the single best combination overall was a support vector machine trained on the most permissive data variant that kept every attribute with at least 40 percent valid values. By contrast, models trained on the strictest variants had as few as 14 to 20 training cases available, and several models performed worse than random guessing under those conditions.
The mechanism behind the pattern appears to be volume. The study found Spearman correlations between training case counts and predictive performance ranging from 0.65 to 0.92 on standardised accuracy, 0.59 to 0.82 on the win–tie–loss measure, and 0.65 to 0.91 on the rank biserial correlation, all statistically significant. In other words, the observed performance improvement was most likely driven by the increased amount of training data, with data quality playing a secondary role. The best-performing models gained more than 14 points in mean standardised accuracy, more than 20 in the win–tie–loss statistic, and more than 0.32 in the effect-size measure when comparing permissive against restrictive variants. The rankings also proved stable: Kendall’s W values were high across random data splits for nearly all models, and data variants that performed well with one model generally performed well with others.
Several interpretations, offered by the author as reasonable explanations rather than directly tested causes, may account for the result. Models trained on very small datasets tend to show high variance, fitting idiosyncrasies of the sample rather than population patterns; adding cases, even noisy ones, gives the model a more diverse set of examples and typically reduces that variance. Lower-rated projects may also cover regions of the feature space, such as particular project types or sizes, that are underrepresented in the small high-quality subset, improving generalisation to new projects. A sensible amount of noise can additionally act as a form of regularisation, discouraging models from placing too much confidence in sample-specific patterns. Notably, variants that kept attributes with as few as 10 to 20 percent valid values often outperformed those keeping only complete attributes, suggesting that with enough cases, the information retained outweighed the loss introduced by simple imputation.
The findings carry important caveats. They derive from a single dataset, the ISBSG 2020 R1 edition, which represents voluntarily submitted projects and may be biased toward those with better documentation. Only 134 projects contained complete satisfaction data, and the target attribute, a Satisfaction Score, was calculated by the study as an unweighted sum of eight four-point satisfaction attributes, an assumption of equal importance that may not match real project priorities, though the internal consistency of the eight dimensions was supported by a Cronbach’s alpha of 0.824. The random cross-validation splits meant training sets sometimes contained projects completed after those in the test sets, a departure from realistic temporal prediction, and model interpretability was deliberately left out of scope. The author frames the conclusions as empirical evidence specific to the ISBSG context rather than a new general principle of machine learning.
Even so, the practical guidance is concrete. For researchers and practitioners working with ISBSG data or datasets with similar characteristics, the study recommends training on relatively large datasets even when they include low-rated cases, retaining attributes with substantial missingness rather than discarding them, and favouring support vector machines or extreme gradient boosting trees, with the support vector machine paired with the permissive variant keeping attributes at 40 percent validity as the first choice. Linear models, neural networks and regression trees performed poorly across almost all variants and are discouraged for this task. Future work, the author suggests, could explore more sophisticated imputation methods, data augmentation to overcome the small case counts, transfer learning from related targets, and interpretability techniques such as SHAP and LIME, which would be essential before deploying such models in practice.
Subject of Research: The trade-off between data quality and quantity in machine learning prediction of user satisfaction in software projects using the ISBSG dataset
Article Title: Quality versus quantity: including imperfect training data improves prediction of user satisfaction in software projects
Article References: Radliński, Ł. (2026). Quality versus quantity: including imperfect training data improves prediction of user satisfaction in software projects. Knowledge and Information Systems, 68(1), Article 295. https://doi.org/10.1007/s10115-026-02898-9
Image Credits: AI Generated
DOI: 10.1007/s10115-026-02898-9
Keywords: machine learning, user satisfaction, software projects, ISBSG dataset, data quality, training data, support vector machines, cross-validation, missing data imputation, predictive modelling, software engineering, data selection
News Source: Denise Maddox. (October 26, 2026). Imperfect but plentiful training data improves prediction of software user satisfaction. Scienmag.
