Cognitive load index ranks programmers, with transfer learning lifting accuracy to 87%

A framework that scores programmers on both technical and non-technical skills, then classifies them into five proficiency ranks using a cognitive load index, has been tested with transfer learning in a study published in Discover Computing. In the study, 190 programmers—140 undergraduate and 50 postgraduate computer science students and research scholars—completed a Google Form assignment containing 15 coding tasks in the C language, covering basic coding, arrays, nested loops, and conditional and logical statements. The tasks required participants to deduce program outputs, fill in blanks, and explain code. Alongside the technical work, each participant completed a self-assessment report covering their understanding of the code, difficulty level, recall, stress levels, emotions, and feelings experienced during the assignment.

The researchers, Karambir Kaur, Ashutosh Mishra, and Palika Chopra, built the framework around a Cognitive Load Index (CLI) that combines two equally weighted components: a Technical Index (TI) and a Non-Technical Index (NTI). The TI is calculated from three technical skills—code interpretation, code debugging, and code construction—weighted 1/6, 2/6, and 3/6 respectively, a weighting the authors align with Bloom’s taxonomy progression from basic understanding to higher-order thinking. The NTI draws on three sub-indices: a Critical Thinking Index (CTI) based on code reasoning ability, performance level, difficulty level, and anxiety level, each weighted 0.25, with difficulty and anxiety entering inversely because higher perceived difficulty and greater anxiety reduce critical thinking; an Emotional State Assessment (ESA) that weights pre-task and post-task emotions at 0.25 each and in-task emotions at 0.5; and a Long-Term Memory (LTM) index covering recall, syntax understanding, logical error tracing, code completion proficiency, and algorithm comprehension, each weighted 0.20. The CLI is then computed as 0.5 times TI plus 0.5 times NTI, and participants are ranked from 1 to 5, with Rank 1 assigned to CLI values between 0.81 and 1.00, described as a very high level of expertise.

After data collection, the team applied min-max normalization because the collected dataset contained negative values across different skills, rescaling all indices to fall between 0 and 1 while maintaining proportional relationships between values. The resulting class distribution across the 190 participants was uneven, a point the authors flag as a limitation that can affect models requiring balanced class distributions. To validate the empirically derived rankings, the researchers turned to transfer learning, a machine learning approach that uses knowledge gained from a pre-training dataset to improve performance on a related fine-tuning dataset. They pre-trained CatBoost and TabNet classifiers on a source dataset from another researcher, which contained technical attributes such as code debugging, code understanding, code design, and code dynamic programming, and non-technical measures including problem-solving, emotions, long-term memory, beliefs, desires, intentions, and commitment. They then fine-tuned the models on their own programmer dataset.

A key methodological detail concerns circular prediction, a risk in expertise classification studies where the target variable is based on the same features used for training. Because participant ranks were created through the CLI-based categorization, the authors trained the transfer learning models exclusively on raw features collected from participants—code interpretation, debugging, construction, perceived difficulty, performance, code reasoning ability, anxiety, emotional states before, during, and after the task, recalling ability, syntax understanding, logical error tracing, code completion proficiency, and algorithm comprehension. Derived indices such as TI, NTI, CTI, ESA, LTM, and CLI were not included as input features; they were used only for framework creation and participant ranking. The models therefore learned relationships directly from the original measures rather than replicating the ranking formulas.

The headline result is a large performance gap between the two classifiers after fine-tuning. The fine-tuned CatBoost model achieved 87% accuracy, a significant improvement over the pre-trained model’s 31%. Precision, recall, and F1 scores for all ranks improved considerably after fine-tuning, with many nearing 1.00. TabNet, which can learn from tabular data via attentive feature selection, performed worse: it achieved 34% accuracy during pre-training, with particularly poor precision and recall for Ranks 1, 2, 4, and 5, and after fine-tuning no improvements were observed, with overall accuracy remaining at just 32%. The authors attribute TabNet’s struggles to its sensitivity to class imbalance and limited data, while crediting CatBoost’s native handling of categorical features through ordered boosting, its regularization techniques that mitigate overfitting, and its robustness with limited data.

Supporting analyses reinforced the picture. A 5-fold stratified cross-validation using the CatBoost classifier produced a mean accuracy of 83.16% with a standard deviation of 6.98%, and an average weighted F1-score of 82.87% with a standard deviation of 7.16%, indicating consistent performance across different data splits. A sensitivity analysis tested two alternative weightings of the CLI—60:40 favoring technical skills and 40:60 favoring non-technical skills—against the original 50:50 scheme. Agreement levels were 88.42% and 85.79% respectively, with rank changes in only 22 participants under the 60:40 setup and 27 under the 40:60 setup, meaning over 85% of participants kept their original ranks even when weights shifted by ±10%. An ablation study compared four feature configurations: technical features alone reached 63.16% accuracy and 62.28% F1-score; non-technical features alone reached 76.84% accuracy and 76.45% F1-score; the combined TI + NTI configuration performed best with 89.47% accuracy and 89.46% F1-score; and all raw features achieved 83.16% accuracy and 82.87% F1-score. The authors interpret this as showing that non-technical features carry significant information about the framework-derived categories and that the aggregated indices capture complementary roles of both skill types while minimizing redundancy.

Individual case examples from the dataset sample illustrate how the ranking works in practice. A Rank 1 participant, described as an exceptional performer, had a CLI of 0.91 with a TI of 0.93 and an NTI of 0.88, excelling in code interpretation, debugging, and construction with strong emotional stability and low anxiety measured at 0.20. A Rank 5 participant, the lowest category, had a CLI of 0.16, a TI of 0.10, and an NTI of 0.21, facing high task difficulty at 0.80 and moderate anxiety at 0.60. Another case shows the framework’s dependence on both components: a participant with a relatively high TI of 0.79 but a much lower NTI of 0.35 received only Rank 3, illustrating that technical ability alone did not produce a top rank without supportive non-technical traits. Boxplots and scatter plots of TI and NTI across ranks showed that both indices decrease progressively from Rank 1 to Rank 5, with a positive correlation between the two indices that is not perfect, since some participants had high TI but only moderate NTI or the reverse.

The authors are explicit that the classification results must be interpreted with care. In their discussion, they note that the Rank 1 to Rank 5 labels were produced using the proposed scoring framework and are based on the same underlying measurements in the input data. Although TI, NTI, and CLI were not directly fed into the model as features, they are related to the input measurements and the framework-derived target labels. The classification performance therefore mainly shows the models’ ability to learn and replicate the categorization scheme, and should not be taken as independent proof that the five categories represent externally validated levels of programmer expertise. The conclusion section repeats this point, stating that the five programmer ranks are based on the scoring framework and its metrics, so the results do not independently validate a programmer’s expertise.

Several further limitations shape the findings. Participant ranks came from a rule-based scoring system based on TI and NTI values, and the weighting factors and rank thresholds were based on literature assumptions and observations rather than expert judgment. The measurement of non-technical attributes such as emotion, stress, and memory retention relied on normalized scales without in-depth psychological profiling. The dataset’s uneven rank distribution underrepresented some categories, and the framework depends on static datasets without accounting for temporal variations in cognitive performance such as fatigue, learning progression, or emotional fluctuations. The sample was also skewed toward individuals with internet access, a willingness to participate in surveys, and available time, and the experiment involved a limited number of participants working on smaller code snippets, so generalizing to larger code snippets may yield different outcomes. Self-reported data on emotions or critical thinking skills could be inaccurate or inconsistent, and unmeasured variables such as prior programming experience, access to resources, or environmental factors were not captured.

Despite these caveats, the authors see practical applications for software engineering education and recruitment. The CLI score could help identify high-performing candidates, pinpoint areas for improvement such as stress management or memory retention, and guide tailored learning or recruitment strategies, with uses including candidate triage into broad proficiency bands, targeted training workshops, adaptive assessment with differentiated exercises, and transparent scoring through the straightforward weighted average. They stress that ethical considerations such as fairness, bias mitigation, and informed consent must be addressed for responsible real-world use. Future research, they write, should validate the framework against independent indicators of programmer expertise not used in the original labeling, including expert or instructor ratings, standardized programming assessments, code quality evaluations, contest performance, debugging success, and task completion outcomes, along with larger and more diverse datasets, physiological measures of cognitive load such as eye-tracking and EEG data, and testing in real-world software development scenarios. The datasets generated and analyzed in the study are not publicly available but are available from the corresponding author on reasonable request.

Subject of Research: Classification of programmer expertise using a cognitive load index and transfer learning

Article Title: Programmer expertise classification using cognitive load and transfer learning

Article References: Kaur, K., Mishra, A., & Chopra, P. (2026). Programmer expertise classification using cognitive load and transfer learning. Discover Computing, 29(1), Article 834. https://doi.org/10.1007/s10791-026-10760-3

Image Credits: AI Generated

DOI: 10.1007/s10791-026-10760-3

Keywords: programmer expertise, cognitive load, transfer learning, CatBoost, TabNet, technical skills, non-technical skills, code comprehension, self-assessment, classification, software engineering education, cross-validation