Explainable Hybrid Machine Learning System Improves Cyber Intrusion Detection in Network Benchmarks

Computer network defenders facing increasingly sophisticated attacks across digital infrastructures and Internet of Things ecosystems may benefit from a newly proposed hybrid framework that couples ensemble machine learning with transparent algorithmic explanations. In a study published in Discover Computing, researchers Asimkiran Dandapat and Bhaskar Mondal introduced an explainable artificial intelligence architecture designed to protect interconnected environments within the emerging Industry 5.0 paradigm. Modern networks, which continuously link smart sensors, smartphones, collaborative robotics, and automated controllers, remain fundamentally vulnerable to novel attack vectors that often bypass traditional perimeter safeguards such as encryption, authentication, and access control lists. While contemporary machine learning and deep learning systems can effectively detect anomalies, they typically operate as opaque black boxes whose automated decision pathways remain inscrutable to human analysts. The newly documented framework resolves this operational vulnerability by systematically integrating adaptive data sampling, statistical feature selection, ensemble classification algorithms, and cooperative game-theoretic explanation models into a unified threat detection and human feedback workflow.

The operational context driving this research stems from the rapid evolution of smart networks and the strict dependability demands of Industry 5.0, where automated machinery and human supervisors must cooperate safely and sustainably. Intrusion detection systems act as continuous network monitors that inspect passing data flows, flag policy violations, and generate automated alerts to intercept unauthorized activities before infrastructure compromises occur. Although complex deep learning models provide high baseline detection capacities, their inability to articulate the underlying technical rationale behind a given classification creates severe friction in operational settings. Furthermore, intrusion detection classifiers frequently suffer from severe bias-variance trade-offs, catastrophic overfitting, and a single-learner instability problem, in which an individual statistical model fails to generalize robustly across unseen traffic profiles or diverse network configurations.

To overcome these systemic vulnerabilities, the research team designed an end-to-end processing pipeline structured across four distinct sequential phases: exploratory data preprocessing, algorithmic feature selection, ensemble training, and explainable performance evaluation with feedback loops. In the initial data preparation stage, the researchers addressed critical data quality issues by identifying and handling missing null entries, zero values, and negative data points across raw flow captures, followed by rigorous mathematical normalization to ensure stable computational processing. Because raw network telemetry contains dozens of redundant attributes that increase computational costs and induce statistical overfitting, the authors mapped linear dependencies using Pearson correlation coefficients. By generating full correlation matrices and visual heatmaps, the investigators established an explicit exclusion threshold: whenever two features demonstrated a correlation coefficient of eighty percent or higher, one feature from the collinear pair was systematically pruned from the dataset, thereby stripping away non-informative statistical noise before model training.

A pervasive barrier in training reliable intrusion detection classifiers is severe class imbalance, as normal network background activity naturally generates millions of benign connection flows while malicious exploits produce comparatively sparse observations. When trained on such unbalanced distributions, algorithms exhibit an acute bias toward the majority benign class, yielding deceptive apparent accuracy while failing to intercept critical minority attacks. To correct this defect without corrupting underlying data distributions, the framework incorporates an Adaptive Synthetic sampling technique known as ADASYN. Rather than replicating existing minority samples uniformly, ADASYN dynamically assesses the distribution of data points and adaptively shifts the classification decision boundary toward the most challenging minority instances. The algorithm achieves this balanced data representation by calculating local density distributions and generating synthetic minority training vectors through targeted linear interpolation, effectively training the ensemble learners to recognize difficult intrusion signatures without succumbing to majority class bias.

Following imbalance correction, the architecture applies independent filter-based feature selection methods to further minimize computational complexity, reduce dimensionality, and eliminate uninformative parameters prior to training. Filter methods evaluate the intrinsic statistical behavior of raw input features completely independently of the downstream classification algorithms, ensuring rapid and scalable computational performance across massive data repositories. The study comprehensively evaluated three statistical filter methods alongside an unfiltered baseline: the categorical Chi-Square contingency test, the ANOVA F-test based on between-group versus within-group variance ratios, and Mutual Information metrics. Mutual Information proved uniquely advantageous because it quantifies non-negative non-linear dependencies and entropy reduction between candidate features and target attack classifications. By retaining only the most informative features, the pipeline accelerates processing throughput while maintaining high descriptive fidelity, preventing classifiers from wasting computational resources on extraneous metrics that do not meaningfully separate malicious intrusions from standard operational communications.

For the core classification engine, the framework leverages ensemble learning paradigms specifically structured to overcome the bias-variance dilemma and single-learner instability. The researchers investigated two robust ensemble architectures: Bootstrap Aggregating, commonly designated as Bagging, and heterogeneous Voting Classifiers. Bagging utilizes homogeneous base learners trained on parallel bootstrap subsets of the training corpus, subsequently aggregating their independent predictions through statistical averaging to dramatically reduce model variance, prevent overfitting, and eliminate the erratic sensitivity seen in individual decision trees. In contrast, the Voting Classifier harnesses heterogeneous learning algorithms whose diverse individual outputs are combined to enhance collective classification accuracy and generalization. The experimental configurations systematically examined both ensemble architectures across varying estimator counts, specifically evaluating ensembles sized at three, four, five, and seven estimators to pinpoint the optimal balance between high detection efficacy and minimal computational overhead.

To transform these complex ensemble predictions into actionable, transparent security intelligence, the architecture integrates a dedicated Explainable Artificial Intelligence layer grounded in Shapley Additive exPlanations, widely known as SHAP. Borrowing mathematical concepts from cooperative game theory, the SHAP framework fairly distributes credit for each classification output across all contributing input variables by computing Shapley values, which reflect the average marginal contribution of each feature across every possible feature permutation. This mathematical approach demystifies complex ensemble logic by producing global summary plots that illustrate overall feature importance rankings based on mean absolute SHAP values, while simultaneously delivering granular local explanations that reveal precisely which metric values drove a specific alert. These explanatory outputs feed directly into an interactive Decision Support Dashboard equipped with structured human-in-the-loop oversight, allowing cybersecurity analysts to evaluate model reasoning, validate findings against domain knowledge, and return corrective feedback to refine model parameters continuously.

The empirical evaluation of this explainable framework was executed across two widely recognized public benchmark environments: the Canadian Institute for Cybersecurity CICIDS2017 dataset and the IoT-focused MQTT-IoT-IDS2020 dataset. The CICIDS2017 corpus contains contemporary multi-stage assaults, protocol records, and diverse Denial of Service profiles captured via CICFlowMeter, comprising 2,830,743 original data records that were refined to 2,827,876 entries after removing corrupted null entries, with 556,556 attack instances set against 2,272,320 benign flows. Correlation pruning condensed its initial 78 features down to 43 highly informative attributes. The MQTT-IoT-IDS2020 benchmark captures telemetry from a virtualized IoT network featuring twelve sensors, an MQTT broker, an IP camera, and an active attacker across five capture files, where correlation filtering streamlined 23 features into 19 informative variables. Evaluating models across both corpora ensured that the resulting performance metrics reflect both enterprise computer network traffic and lightweight machine-to-machine IoT messaging environments.

Experimental testing demonstrated that the proposed hybrid architecture consistently outperformed existing intrusion detection models across standard validation metrics, including accuracy, precision, recall, and F1 scores computed across both benign and malicious traffic classes. On the CICIDS2017 benchmark, the proposed framework achieved an overall classification accuracy of 98.67 percent and an F1 score of 97.86 percent, while reaching 98.35 percent accuracy and an F1 score of 97.57 percent on the MQTT-IoT-IDS2020 dataset. When comparing the two core ensemble methods, Bagging proved superior in overall accuracy, surpassing the Voting Classifier by 4.73 percentage points on CICIDS2017 (98.48 percent versus 93.75 percent) and by 1.18 percentage points on MQTT-IoT-IDS2020 (98.43 percent versus 97.25 percent). Bagging also secured higher attack recall on CICIDS2017 at 97.25 percent compared to 94.11 percent for Voting, whereas Voting registered a fractional recall advantage on the MQTT data at 97.45 percent versus 97.36 percent.

The experimental comparisons further revealed that Mutual Information consistently outperformed rival feature selection filters, sustaining testing accuracies above 98.2 percent with high consistency and a negligible generalization gap on CICIDS2017. Conversely, conventional filters like ANOVA struggled severely, failing to identify roughly 18 percent of active intrusions on CICIDS2017 with an attack recall of only 82.31 percent, and similarly faltering on the MQTT dataset with an attack recall of 83.14 percent. Rigorous statistical validation utilizing the Wilcoxon Signed-Rank Test confirmed that the observed performance gains were statistically significant, consistently meeting confidence thresholds with probability values below 0.05 across all experimental conditions, highlighted by Bagging on CICIDS2017 reaching a probability value below 0.001. The authors noted that while feature selection remains indispensable for large feature dimensions, compact datasets can occasionally bypass selection without substantial degradation. They also cautioned that deploying highly granular explainability layers introduces incremental architectural complexity and potential data privacy risks in sensitive environments such as healthcare, identifying balanced privacy-preserving explanations as a primary objective for future investigation.

Subject of Research: Explainable artificial intelligence and ensemble learning for network intrusion detection

Article Title: SHAP based hybrid intrusion detection scheme using ADASYN, filter method, and ensemble learning for IoT and computer network

Article References: Dandapat, A., & Mondal, B. (2026). SHAP based hybrid intrusion detection scheme using ADASYN, filter method, and ensemble learning for IoT and computer network. Discover Computing, 29(1), Article 838. https://doi.org/10.1007/s10791-026-10779-6

Image Credits: AI Generated

DOI: 10.1007/s10791-026-10779-6

Keywords: intrusion detection system, explainable artificial intelligence, machine learning, SHAP, ADASYN, ensemble learning, Internet of Things, network security, cybersecurity, feature selection, based, hybrid