Conformal prediction triage exposes hidden coverage gaps in intrusion alerts

High benchmark accuracy does not tell a security operations center which alerts to trust, and it says nothing about attacks that never appeared in training. A new study in Discover Informatics puts that operational gap under the microscope by auditing conformal prediction, a statistical framework that attaches mathematically guaranteed coverage to a classifier’s outputs, as an analyst-triage layer on two widely used intrusion detection benchmarks. The audit, led by Sesank Vardhan Arepalli, Ujval Sai Gopal Irrinki, Arun Morampudi, and Rakesh Kumar Surapani, ran 15 leave-one-attack-family-out experiments on UNSW-NB15 and CIC-IDS2017 and compared probability-space novelty scores against feature-space distance scores across every held-out family.

The central finding is starkly family dependent. Four attack families reached a probability-space area under the receiver operating characteristic curve of at least 0.70 while the nearest-neighbor distance score fell to 0.45 or below, including CIC-IDS2017 web attacks at 0.966 against 0.298, UNSW-NB15 Backdoor at 0.860 against 0.417, Shellcode at 0.909 against 0.307, and Reconnaissance at 0.759 against 0.364. Port scanning reversed the ordering entirely, scoring 0.166 for the probability measure against 0.812 for distance. The authors attribute this reversal to the axis-aligned tree ensembles used as base classifiers, which cannot extrapolate and assign near-certain probabilities to unbounded regions of feature space, so a flow with extreme values on a few features lands in a pure leaf and receives a confident label precisely because the model has no representation of distance.

Neither ordering is stable across capture environments. Scanning activity appears in both datasets, as Port Scanning in CIC-IDS2017 and, by the dataset authors’ own definitions, within Analysis and Reconnaissance in UNSW-NB15, and the two scores swap roles between them. The benign-only isolation forest, the conventional unsupervised baseline, was weaker overall, reaching 0.75 or above on only two of the 15 families and inverting on CIC-IDS2017 port scanning at 0.267. A matched forest fitted on all known classes achieved a mean AUROC of 0.538 across the held-out families, compared with 0.587 for the benign-only variant, confirming that no single score delivers a uniform warning.

The triage layer itself works by converting a base classifier’s estimated class probabilities into a prediction set whose cardinality drives the operational decision. A singleton set is auto-dispatched without human review, a set of two or more classes is escalated as ambiguous, and an empty set is also escalated. Under split conformal prediction, a held-out calibration sample determines a quantile at a chosen miscoverage level, and the resulting set carries a finite-sample marginal coverage guarantee under exchangeability, with no distributional assumptions and no change to the model. The study evaluated three constructions: the marginal least-ambiguous set-valued classifier, the randomized adaptive prediction set, and the class-conditional or Mondrian construction that computes a separate quantile per class.

Marginal coverage matched the nominal 0.900 target in all conditions tested, but the per-class picture told a different story. Under marginal calibration on UNSW-NB15, coverage fell to 0.295 on denial of service, 0.411 on analysis, 0.417 on worms, and 0.475 on backdoor, with six of ten classes below 0.85, while the aggregate figure read 0.900. On CIC-IDS2017, the same construction yielded 0.568 for port scanning. The mechanism follows from theory: a single global quantile is dominated by the classes supplying most calibration points, so rare and hard classes absorb the miscoverage budget of the entire distribution. Class-conditional calibration repaired this, raising every seed-averaged class coverage to at least 0.894 on both datasets, including a class with only 35 calibration examples.

The repair has a cost that depends on the base classifier rather than the dataset. For gradient boosting on UNSW-NB15, class-conditional calibration raised the mean prediction set size to 2.90 and collapsed Rule B auto-dispatch to 0.3 percent, so that configuration offers per-class coverage or useful automation but not both. The random forest on the same data attained comparable per-class coverage, worst class 0.895, while dispatching 70.4 percent. On CIC-IDS2017, the random forest again paid little, at 90.9 percent, while gradient boosting was bimodal across seeds, with three of five seeds dispatching above 91 percent and two collapsing to 0.2 percent. The authors caution that this is a material model-specific efficiency limitation that should be checked on every refit.

Escalation is quantified as the fraction of classifier errors sent to an operator for a given review budget, a quantity that differs by a factor of two between datasets and cannot be predicted from accuracy. Base-error capture reached 98.2 percent on UNSW-NB15 with gradient boosting and 84.1 percent on CIC-IDS2017 with the random forest under class-conditional calibration. Between 73.2 and 100 percent of the base classifier’s missed intrusions, attacks it would have labelled benign, were routed to a human under class-conditional or adaptive calibration on UNSW-NB15, and between 94.3 and 98.7 percent on CIC-IDS2017. What remained auto-dispatched was dominated by benign flows labelled as attacks and by attacks attributed to the wrong family, costing analyst time and misrouting rather than a missed compromise.

Against a tuned probability threshold that targets the same dispatch and error rates, the conformal layer often achieved similar or lower selective risk, but only the conformal constructions supplied finite-sample coverage control. Against a per-class empirical quantile that differs only in omitting the finite-sample correction, the conformal version lifted worst-class coverage from between 0.858 and 0.899 to between 0.894 and 0.901, at essentially unchanged dispatch. The gap was largest where classes were smallest, which is where the adjustment is designed to matter. A deployment that imposes a minimum-class-size fallback, a common convenience, forfeits the guarantee for precisely the rare classes the construction protects: on UNSW-NB15, setting the threshold at 50, which excludes the 35 calibration examples of the Worms class, collapsed worst-class coverage to 0.417 while the marginal figure remained unchanged at 0.899.

A temporal partition of CIC-IDS2017 tested the layer where training precedes evaluation, fitting on Monday to Wednesday flows and calibrating and testing on Thursday and Friday, within which 71.7 percent of flows belonged to families entirely absent from training. For the classes the model had seen, it retained an accuracy of 0.999 and coverage of 0.902 against the 0.900 target. The authors stress that this figure says nothing about detection or coverage of the absent families, since no coverage statement of any kind applied there, and that stale-calibration settings require conformal variants designed for covariate shifts and non-exchangeable data.

An exploratory attribution analysis using Shapley additive explanations on 1,000 escalated and 1,000 auto-dispatched flows found that mean absolute attribution profiles were nearly identical across the 12 most influential features, providing no evidence that attributions explain the escalation decision. This is consistent with escalation being a property of the geometry of the predicted probability vector rather than of the features driving it. The study positions conformal prediction as a coverage control mechanism rather than a universal novelty detector or automation advantage, and recommends class-conditional calibration without a minimum-class-size fallback wherever per-class coverage is a hard requirement, with per-class coverage reported alongside any marginal figure and efficiency verified for the chosen base classifier.

Subject of Research: Conformal prediction as an analyst-triage layer for network intrusion detection, including novelty scoring of unseen attack families and per-class coverage control on UNSW-NB15 and CIC-IDS2017.

Article Title: Conformal prediction for network intrusion triage and analysis of unseen attack families

Article References: Arepalli, S. V., Irrinki, U. S. G., Morampudi, A., & Surapani, R. K. (2026). Conformal prediction for network intrusion triage and analysis of unseen attack families. Discover Informatics, 1(1), Article 33. https://doi.org/10.1007/s44564-026-00033-y

Image Credits: AI Generated

DOI: 10.1007/s44564-026-00033-y

Keywords: conformal prediction, network intrusion detection, security operations center, UNSW-NB15, CIC-IDS2017, novelty detection, class-conditional calibration, coverage guarantee, attack families, machine learning, alert triage, prediction sets