Hybrid AI framework trims redundant features to speed intrusion detection benchmarks

Researchers have reported that a framework combining correlation-based feature pruning with a resource-aware adaptive learning layer can improve both predictive performance and computational efficiency on intrusion-detection benchmark datasets. The study, published in Discover Artificial Intelligence, describes a Complexity-Aware Adaptive Hybrid AI–ML framework that was evaluated against conventional and static feature-selection baselines on three publicly available cybersecurity benchmarks: NSL-KDD, UNSW-NB15 and CICIDS2017. The authors are careful to restrict their claims to these evaluated benchmarks and the documented computational conditions, rather than generalizing to all big-data domains.

The central idea of the work is coordination rather than invention. Correlation-based redundancy pruning and adaptive learning mechanisms have each been investigated independently in prior research, but the authors argue that existing approaches generally optimize feature redundancy and learning behavior as separate stages. In the proposed framework, the artificial intelligence component performs redundancy-aware feature analysis and adaptive resource-aware decision making, while the machine-learning component trains predictive models on the optimized feature space with dynamically adjusted learning parameters. The contribution, as the paper states explicitly, lies in the unified coordination of these complementary components rather than in proposing a completely new machine-learning algorithm or artificial intelligence paradigm.

The pipeline proceeds in four stages. Raw data first undergoes preprocessing, including cleaning, treatment of missing values, categorical encoding and normalization of numerical attributes. The preprocessed data then passes to a redundancy-aware feature-pruning module, where pairwise Pearson correlation coefficients are computed among numerical feature representations to quantify linear redundancy between feature pairs. A correlation threshold of 0.90 was adopted as a conservative criterion that removes only strongly correlated features while minimizing the risk of discarding informative variables; one representative feature from each highly correlated pair was retained. The reduced feature set is then fed into a resource-aware adaptive learning engine, which dynamically adjusts training parameters such as learning rate, batch size and iteration control based on available computational resources including CPU usage, memory availability and processing load. Finally, an optimized learning model is produced.

The adaptive layer is governed by a mathematical control relationship in which the learning rate at each iteration is scaled by a factor that depends on system resource utilization and feature-space complexity, with two adaptive scaling coefficients controlling the sensitivity to each. The authors are notably candid about the limits of their archived experimental record: the exact numerical values of these scaling coefficients, the monitoring interval and the resource thresholds were not preserved, so the equation is presented as the mathematical specification of the proposed control relationship rather than as a report of verified coefficient values. The adaptive control layer uses feature-space dimensionality as a complexity proxy and processor and memory measurements as resource indicators. Importantly, feature pruning itself is performed once before model training; the adaptive stage does not dynamically change the selected feature subset during an individual training run, and the authors removed any broader claim of runtime feature-pruning adaptation because the original experimental record does not provide sufficient evidence for such a mechanism.

Dataset-specific pruning results illustrate how the procedure operated in practice. For the NSL-KDD dataset, the original feature space contained 41 predictor variables, and Pearson correlation analysis with the 0.90 threshold identified seven highly correlated features, leaving 34 features for model training. For UNSW-NB15, the original feature space contained 34 predictor variables after preprocessing and categorical encoding, and the same analysis identified 10 highly correlated features, resulting in a reduced set of 24 features. For CICIDS2017, the same preprocessing and pruning protocol was applied, but dataset-specific pruning counts were not recorded in sufficient detail in the source study and were not reconstructed, an omission the authors describe as a deliberate avoidance of inventing a reduction ratio.

All experiments were conducted in the Google Colaboratory cloud environment using Python 3.12.13, with NumPy 2.0.2, Pandas 2.2.2, Scikit-learn 1.6.1 and XGBoost 3.3.0. The runtime was configured with an Intel Xeon CPU at 2.20 GHz providing two virtual CPU cores and 12 GB of system memory, operating on an x86_64 architecture under a KVM-based virtualized environment. The framework was compared against three baselines: a conventional Stochastic Gradient Descent classifier trained on the complete feature set without redundancy pruning, a Random Forest classifier combined with correlation-based feature selection representing a conventional feature-selection approach, and an XGBoost classifier using the same correlation-pruned feature subset but without the resource-aware adaptive learning mechanism, representing a static hybrid optimization approach. All models were trained and evaluated with identical preprocessing, partitions and evaluation metrics, and hyperparameters were selected through preliminary empirical tuning and then kept fixed.

The reported aggregate results indicate measurable gains for the proposed configuration. The aggregate table reports 94.80 percent accuracy for the proposed framework versus 91.30 percent for the static hybrid configuration, an absolute difference of 3.50 percentage points, or approximately 3.83 percent relative to the static value. The authors explicitly caution that these aggregate figures should not be interpreted as dataset-wise repeated-run estimates, because such statistics were not recorded in the original study, and they avoid claims of statistical significance that cannot be supported by the available experimental record. On memory, the reported aggregate peak values were 1,850 megabytes for the proposed configuration and 2,300 megabytes for the static hybrid configuration, indicating lower observed memory use in the reported experiment, though the authors note that these values do not by themselves establish performance in distributed deployments.

Training-time analysis attributed the efficiency gains primarily to redundancy-aware feature pruning and adaptive parameter optimization, which the authors say collectively minimize unnecessary computations and improve convergence efficiency. A component-wise ablation analysis was also conducted, in which the complete framework was compared with a reduced configuration in which correlation-based redundancy-aware feature pruning was omitted while maintaining identical preprocessing, partitions, learning settings and evaluation metrics. According to the paper, removing the pruning component increased computational overhead and slightly reduced predictive performance because redundant features remained in the learning process, indicating that redundancy-aware feature optimization contributes to improved computational efficiency while preserving predictive capability. The authors also provide a computational complexity analysis: pairwise correlation computation requires quadratic complexity with respect to the number of input features, but because this analysis is executed once before training, subsequent adaptive learning operates on a reduced feature subset, lowering training cost and memory requirements relative to processing the complete feature space.

The study is equally explicit about its limitations. The Pearson correlation-based redundancy analysis primarily captures linear relationships among features and may not fully identify complex nonlinear dependencies, and the authors acknowledge that correlation-based pruning may not capture higher-order interactions among features that become informative only when considered jointly. The experimental evaluation is limited to benchmark datasets and supervised classification tasks; the framework has not yet been evaluated on streaming data, distributed computing environments or large-scale real-time applications. The adaptive learning mechanism improved training efficiency under the evaluated computational resource conditions, but broader suitability for distributed, cloud-edge or continuously varying production environments should be regarded as a prospective application rather than an experimentally established result. Privacy-preserving optimization is likewise identified as future work rather than an experimentally evaluated capability.

Future directions outlined in the paper include nonlinear redundancy analysis, dynamic feature selection, online adaptive learning, distributed implementations, automated redundancy-threshold selection, explainable artificial intelligence techniques to improve model interpretability, federated learning for privacy-preserving distributed optimization, and validation using additional real-world big data applications. The authors also plan to investigate adaptive threshold selection based on feature-space characteristics and resource availability, as well as interaction-aware feature selection methods such as mutual information analysis and graph-based feature dependency modeling. For now, the documented contribution is a reproducible workflow, applied consistently across three intrusion-detection benchmarks under a fixed correlation threshold and identical experimental conditions, in which coordinated feature-space reduction and resource-aware learning control delivered competitive classification performance with reduced training time and memory consumption in a CPU-based environment.

Subject of Research: A hybrid AI–ML framework that combines correlation-based feature pruning with resource-aware adaptive learning for intrusion detection benchmarks

Article Title: A complexity-aware adaptive hybrid AI–ML framework for redundancy-aware feature optimization in intrusion detection benchmarks

Article References: Krishna, P., Dekka, S., Rafi, M., & Sai, D. M. (2026). A complexity-aware adaptive hybrid AI–ML framework for redundancy-aware feature optimization in intrusion detection benchmarks. Discover Artificial Intelligence, 6(1), Article 1667. https://doi.org/10.1007/s44163-026-02518-z

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02518-z

Keywords: intrusion detection, feature selection, machine learning, adaptive learning, data redundancy, NSL-KDD, UNSW-NB15, CICIDS2017, computational efficiency, XGBoost, Pearson correlation, resource-aware computing