Machine learning benchmark flags hidden misses in road-accident prediction

Road traffic accidents remain one of the most persistent public-safety problems of modern transportation, and smart-city planners have increasingly turned to machine learning to identify when and where an accident is likely to occur.

The study, conducted by Areen Arabiat of the Department of Communications and Computer Engineering and Hamza Abu Owida of the Department of Medical Engineering at Al-Ahliyya Amman University in Jordan, used the Transportation and Accident Prediction Dataset, a publicly available synthetic dataset hosted on Kaggle. The dataset contains 5,500 traffic observations described by 17 attributes, covering location, time, traffic conditions, weather, road surface state, identifiers and outcome variables. Thirteen of those attributes served as predictors, including timestamp, latitude and longitude, road identifier, vehicle count, average speed, vehicle type, traffic density, weather, visibility distance in metres, road condition, temperature and humidity.

A central methodological decision in the work concerned which variables were kept out of the models. Accident_Occurred, a binary yes-or-no field, was defined as the primary prediction target. Record_ID was excluded because it is a mere identifier, while Alert_Generated and Accident_Severity were treated as auxiliary outcome variables and removed from the predictor set to prevent target leakage. The authors describe this separation as the mechanism that keeps outcome-related information from being unintentionally used as a predictor.

Six classifiers were evaluated under an identical experimental setup implemented in the Orange3 data-mining environment: K-nearest neighbors (KNN), AdaBoost, the CN2 Rule Inducer, support vector machine (SVM), Random Forest (RF) and Naive Bayes (NB). Each algorithm represents a different family of supervised learning. Random Forest builds an ensemble of decision trees trained on bootstrap samples with random subsets of predictors at each split, and classifies by majority voting across the fitted trees. AdaBoost sequentially combines weak learners, reweighting observations that earlier learners misclassified so that later learners focus on hard cases, a strategy that improves discrimination but can increase sensitivity to noise and outliers.

KNN, by contrast, is a non-parametric, instance-based method that assigns a new observation to the class most frequent among its k nearest neighbors under a chosen distance measure. SVM seeks a decision boundary with the maximum margin between the two classes, with the observations closest to the boundary, the support vectors, defining the separating hyperplane, and it can represent nonlinear boundaries through kernel functions. The CN2 Rule Inducer learns human-readable IF-THEN rules through sequential covering, offering direct interpretability at the cost of a greedy search that does not guarantee a globally optimal rule set. Naive Bayes estimates posterior class probabilities from class priors and feature likelihoods under a conditional-independence assumption that is simplified relative to real data but often remains computationally efficient and competitive.

Evaluation used 10-fold cross-validation: the data were divided into ten approximately equal folds, nine folds trained the model and the remaining fold tested it, and the process repeated until every fold had served exactly once as the held-out test set. Predictions from the held-out folds were aggregated across the full dataset, and any data-dependent preprocessing, such as encoding categorical variables and scaling numerical features for magnitude-sensitive algorithms, was estimated on the training portion of each fold and applied to its held-out portion to avoid leakage. Performance was summarized with true and false positive and negative counts, accuracy with 95 percent Wilson confidence intervals, positive-class precision, recall, specificity, F1-score and balanced accuracy, with observations labeled Yes for Accident_Occurred treated as the positive class.

The reason for looking beyond accuracy became clear in the results. Only 575 of the 5,500 observations, or 10.45 percent, belonged to the positive accident class, a substantial imbalance. KNN produced 4,924 true negatives, one false positive, 574 false negatives and a single true positive, yielding 89.55 percent accuracy but a positive-class recall of just 0.17 percent. Its apparently respectable score came almost entirely from predicting the majority non-accident class. Among the non-KNN models, AdaBoost achieved the highest recall at 78.09 percent, with 94.67 percent overall accuracy, 449 true positives, 126 false negatives and 167 false positives that pulled its precision down to 72.89 percent.

The other four models clustered tightly. CN2 reached 96.16 percent accuracy with 84.73 percent precision and 77.22 percent recall. SVM achieved 97.40 percent accuracy, 97.37 percent precision and the same 77.22 percent recall. Random Forest and Naive Bayes tied for the highest overall accuracy at 97.62 percent, with 95 percent Wilson confidence intervals of 97.18 to 97.99 percent, and both produced zero false positives, giving perfect specificity of 100 percent. Yet each missed 131 of the 575 positive cases, leaving recall at 77.22 percent and balanced accuracy at 88.61 percent. In a safety application, the authors note, a false negative means the classifier fails to flag an accident that has occurred, which may be more consequential than a false alarm, so recall, specificity, F1-score, balanced accuracy and the relative costs of the two error types should weigh in model selection, alongside probability-based measures such as PR-AUC where available.

The authors are explicit that these numbers should not be read as a verdict on any algorithm’s fitness for real roads. Because the dataset is synthetic, its feature distributions, noise characteristics and logical relationships may not fully represent operational transportation systems, and the results constitute a controlled benchmark rather than evidence of deployment readiness. The limitations section adds that the archived materials lack complete Orange3 version information, full learner hyperparameter settings, probability scores and record-level prediction vectors, which means hard-label confusion matrices alone cannot support PR-AUC estimation, calibration analysis or paired tests such as McNemar’s test. Identical hard-label confusion matrices, the authors point out, also do not establish that two models behave identically at the probability level, something only a fully reproducible rerun retaining per-instance predictions could assess.

For future work, the study calls for replicating the six-classifier benchmark while preserving the full workflow configuration, exporting class-level predictions and per-instance probabilities to enable PR-AUC reporting, calibration and threshold analysis, and paired statistical comparisons, and then validating externally on real-world traffic data from connected vehicles, roadside units, traffic cameras, weather services and IoT sensors. Temporal and geographic holdout designs are recommended to test robustness to distribution shifts, class-imbalance methods such as cost-sensitive learning or resampling should be applied within training folds only, and prediction drivers could be probed with Random Forest importance, permutation importance and interpretable CN2 rules. The broader lesson for smart-city safety analytics, however, is already on the table: when accidents are the rare class, a headline accuracy figure can conceal exactly the failures that matter most, and class-sensitive evaluation is not optional.

Subject of Research: A comparative machine-learning benchmark for binary road-accident prediction using a synthetic transportation dataset

Article Title: A data-driven supervised machine learning study for road-accident prediction in smart transportation operations and planning in smart cities

Article References: Arabiat, A., & Owida, H. A. (2026). A data-driven supervised machine learning study for road-accident prediction in smart transportation operations and planning in smart cities. Frontiers in Future Transportation, 7, Article 1923312. https://doi.org/10.3389/ffutr.2026.1923312

Image Credits: AI Generated

DOI: 10.3389/ffutr.2026.1923312

Keywords: machine learning, road accidents, smart cities, class imbalance, random forest, cross-validation, intelligent transportation systems, synthetic data, benchmark study, traffic safety, Naive Bayes, AdaBoost