Explainable Deep Learning Predicts Digital Literacy Behavioral Proxies in Online Learners

A recent educational data mining investigation has demonstrated that combining recurrent neural networks with attention mechanisms enables predictive models to identify digital literacy behavioral patterns from online course interaction logs. In a study published in Discover Artificial Intelligence, researcher Jie Zhu introduced an explainable learning analytics framework based on Bidirectional Long Short-Term Memory networks and an adaptive attention mechanism to classify learners into behavioral proxy categories. Using interaction traces extracted from the Open University Learning Analytics Dataset, the deep sequential architecture attained an overall classification accuracy of 95.8 percent and an area under the receiver operating characteristic curve of 97.2 percent across three behavioral activity tiers. The findings show that chronological engagement dynamics recorded within learning management systems provide informative signals regarding how students navigate digital instructional environments, while post-hoc explainability methods illuminate the specific actions that drive model classifications.

As digital platforms and web-based instruction become central components of higher education, institutional interest in tracking and supporting digital literacy competencies has expanded considerably. Conventional educational prediction systems deployed in virtual learning environments have historically concentrated on forecasting academic attrition or final course marks, typically relying on traditional machine learning algorithms such as decision trees, random forests, and support vector machines. While these baseline approaches can assess aggregate activity summaries, they treat distinct learner interactions as independent, static observations rather than chronological sequences. Because student interactions within learning platforms evolve over weeks of study, standard models struggle to capture temporal dependencies. Zhu sought to determine whether modeling student clicks, assessment records, and resource requests as ordered sequences could more accurately reflect ongoing engagement patterns and provide actionable insights into digital competencies.

To establish a structured target for sequential prediction in the absence of standardized testing benchmarks, the researcher formulated the Media Digital Literacy Analytics framework, known as MDLA. Because the Open University dataset does not contain independently administered digital literacy examination scores or external survey measurements, the study operationalized digital literacy through observable behaviors recorded in virtual learning environments. The MDLA representation maps platform activities across multiple competency dimensions, including information evaluation, digital ethics, content creation, online collaboration, and resource exploration. For example, assessment completion and resource access were linked to information evaluation, while discussion forum participation reflected digital communication and collaborative engagement. Learners were subsequently grouped into Low, Medium, and High behavioral proxy categories based on these composite profiles, providing standardized output classes for supervised classification.

A critical methodological priority within the experimental workflow involved preventing target circularity and data leakage during feature generation and model evaluation. Because the Low, Medium, and High proxy categories were derived directly from learner behaviors, feeding those same composite dimensions into the predictive models would produce artificial inflation of classification performance. To eliminate circularity, the specific MDLA variables utilized to construct the target categories were removed from the predictor matrix prior to model training. The predictive algorithms were supplied exclusively with independently prepared contextual demographic variables and selected raw behavioral features, such as cumulative click counts, assessment submission dates, login regularity, and resource access diversity. Furthermore, all data scaling parameters and normalizations were calculated strictly on the training partition to ensure that validation and test sets remained completely unexposed during parameter optimization.

The predictive architecture developed for the investigation incorporates five sequential components designed to capture and interpret learner behavior over time. The pipeline begins by transforming preprocessed, normalized interaction logs into chronological state vectors spanning consecutive learning intervals. These input sequences are fed into a Bidirectional Long Short-Term Memory layer consisting of 128 neurons, which processes interactions in both forward and reverse directions to capture dependencies connecting past behaviors with subsequent platform activities. The resulting bidirectional hidden states are then weighted by an adaptive attention mechanism that automatically computes importance scores for individual time steps across the course timeline. An attention-guided context vector is subsequently forwarded through a dense feedforward layer of 64 neurons with rectified linear unit activation, before a final Softmax output layer estimates the posterior probabilities for each behavioral proxy category.

Model training and evaluation were carried out in an isolated experimental setup using an 80:10:10 split for training, validation, and testing partitions. Optimization relied on the Adam algorithm with a fixed learning rate of 0.001, categorical cross-entropy loss, a batch size of 32, and dropout regularization set at 0.30 to suppress co-adaptation. Across the held-out test sample of 254 learners, the proposed architecture achieved a precision of 95.1 percent, a recall of 94.7 percent, and an F1-score of 94.9 percent, alongside the reported 95.8 percent accuracy and 97.2 percent ROC-AUC. When compared directly with standard baseline classifiers evaluated on the identical test split, the attention-augmented bidirectional network demonstrated superior descriptive performance over Random Forest, XGBoost, Multi-Layer Perceptrons, Convolutional Neural Networks, standard unidirectional LSTMs, and standalone BiLSTM configurations.

Confusion matrix and ablation analyses provided granular detail regarding how individual components and classification boundaries functioned during the test evaluation. The confusion matrix revealed that classification errors were predominantly confined to adjacent behavioral categories, such as misclassifications between Low and Medium or between Medium and High profiles, rather than extreme misclassifications. In the component ablation experiments, removing the attention mechanism reduced classification accuracy from 95.8 percent to 93.1 percent and lowered ROC-AUC from 97.2 percent to 94.4 percent. Reverting from bidirectional recurrent processing to a unidirectional LSTM architecture further reduced accuracy to 91.2 percent and ROC-AUC to 92.7 percent. These comparative decrements indicate that both dual-direction temporal contextualization and adaptive step weighting contributed meaningfully to the observed predictive capabilities of the complete model under the tested parameters.

To provide model interpretability alongside numerical predictions, the research integrated SHapley Additive exPlanations, known as SHAP, with temporal attention visualization. The post-hoc SHAP analysis evaluated the global importance of individual input attributes by calculating their average marginal contributions based on cooperative game theory. Engagement duration emerged as the most influential feature with an importance value of 0.146, followed by assessment performance at 0.112, resource usage volume at 0.098, forum discussion participation at 0.076, interaction frequency at 0.064, and access regularity at 0.052. Simultaneously, the attention visualization module examined how importance weights were allocated across ten weekly learning windows. The resulting temporal heatmaps demonstrated that the model assigned disproportionately higher weights to activities occurring during the sixth and eighth instructional weeks, identifying critical phases in the academic term where student interaction patterns most strongly differentiated behavioral categories.

Notwithstanding the high classification metrics reported in the evaluation, several substantial limitations constrain the interpretation and generalizability of the findings. Most importantly, the Open University Learning Analytics Dataset does not include standardized digital literacy measurements, meaning the target categories represent behavioral activity proxies derived from learning management platform records rather than validated cognitive or technical competencies. Additionally, while the conceptual framework was formulated with media education competencies in mind, the open dataset records could not verify whether enrolled participants were enrolled in media-specific curricula, restricting direct claims regarding media students. Furthermore, all reported performance figures stem from a single experimental execution with a fixed random seed, meaning standard deviations, confidence intervals, and formal tests of statistical significance could not be calculated.

Future investigations will need to focus on resolving these methodological boundaries before such systems can inform actual pedagogical interventions. Zhu emphasized that independent external validation represents a necessary step, requiring researchers to deploy the framework alongside standardized, independently administered digital literacy assessments to evaluate criterion and construct validity. Further research is also required to test whether the sequential modeling architecture generalizes across diverse institutional learning platforms, differing subject disciplines, and varied student demographics. Exploring transformer-based sequence encoders and conducting multi-run replications with formal statistical hypothesis testing will help clarify whether attention-augmented deep learning can reliably support instructors in identifying students who require targeted technological guidance in digital learning environments.

Subject of Research: Explainable deep learning and learning analytics for behavioral digital literacy modeling in virtual learning environments

Article Title: Deep learning–driven learning analytics for enhancing digital literacy skills of media students using an explainable BiLSTM-attention framework

Article References: Zhu, J. (2026). Deep learning–driven learning analytics for enhancing digital literacy skills of media students using an explainable BiLSTM-attention framework. Discover Artificial Intelligence, 6(1), Article 1830. https://doi.org/10.1007/s44163-026-02643-9

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02643-9

Keywords: Learning Analytics, Digital Literacy, BiLSTM Network, Attention Mechanism, SHAP Explainability, Open University Learning Analytics Dataset, Educational Data Mining, Virtual Learning Environments, Behavioral Modeling, Predictive Analytics, Sequential Modeling, Higher Education Technology, Feature Engineering, Ablation Analysis, Student Engagement Tracking, Model Interpretability