Machine Learning Models Show Modest Predictive Accuracy for Logistics Delivery Disruptions

A recent investigation into logistics performance has demonstrated that standard machine learning algorithms achieve modest improvements over trivial baselines when predicting supply chain delivery disruptions. Published in the journal Discover Analytics, the investigation operationalized supply chain resilience not as an unobserved or latent organizational capability, but strictly as an observable delivery performance outcome under operational disruption. Using a publicly available secondary logistics dataset containing thousands of order records, the researcher evaluated the capacity of supervised decision-tree ensembles to classify shipments into early, on-time, or delayed categories. The empirical results revealed that while predictive tools can successfully capture specific delay-risk patterns across diverse shipments, their discriminative performance remains constrained by significant overlap among delivery categories and underlying data limitations, reinforcing that such models should function as exploratory screening prototypes rather than autonomous operational decision-making engines.

The study was situated within an operational environment where global logistics networks encounter increasing disruption frequency, a vulnerability acutely exposed during the COVID-19 pandemic. Although artificial intelligence has been widely deployed across broader supply chain operations, previous empirical literature has concentrated predominantly on upstream demand forecasting and inventory optimization rather than downstream delivery disruption detection. Prior research had established that predictive tools could assist logistics managers in transitioning from reactive post-hoc mitigations to proactive scheduling interventions. However, empirical assessments systematically benchmarking classification algorithms on multi-class delivery outcomes, while quantifying the predictive utility of observable logistics variables, remained scarce. The current study was structured to address this empirical gap by addressing three core research questions: how effectively artificial intelligence models can predict delivery disruptions, which operational variables carry the strongest predictive signals, and how distinct shipping strategies and geographical characteristics associate with observed delivery performance patterns.

To explore these research questions, the investigation examined a secondary logistics dataset hosted on Kaggle entitled Logistics Supply Chain Real-world Data, uploaded by an individual contributor under Open Database terms. The dataset comprised order and shipping timestamps, order status, item quantities, total monetary values, geographic markers, and service levels across global markets. The original raw dataset contained 15,549 records with raw delivery duration spans extending from minus 1429 days to positive 1430 days, reflecting substantial recording anomalies inherent to uncurated operational systems. To ensure temporal validity, records exhibiting implausible delivery durations outside an acceptable boundary of minus 30 to 365 days were systematically removed. Because the original organizational source, collection protocol, and external verification mechanisms could not be independently authenticated, the study maintained a strictly exploratory posture, explicitly disclaiming broad empirical generalizability.

The dependent target variable was formulated as a three-class delivery outcome label distinguishing early arrivals encoded as negative one, on-time deliveries encoded as zero, and delayed deliveries encoded as positive one. In the cleaned data, delayed shipments constituted the majority class, representing 57.7 percent of training cases, while early deliveries accounted for 22.8 percent and on-time shipments comprised 19.5 percent. The independent predictor set included shipping mode, market region, order region, order item quantity, order total monetary amount, order status, and derived temporal intervals. Delivery duration was computed directly from the recorded shipping and order timestamps. Methodologically, the author emphasized that because delivery duration was derived from the very temporal fields utilized to construct the delivery outcome categories, it carried target-adjacent properties and potential data leakage characteristics. Consequently, its apparent predictive contribution was interpreted with strict caution rather than treated as an independent operational driver.

Data preprocessing incorporated categorical encoding and feature evaluation designed to preserve algorithmic integrity. Categorical service tiers were label-encoded across four shipping modes, spanning Standard Class, First Class, Second Class, and Same Day shipping, alongside five aggregated market regions comprising Africa, Europe, Latin America, Pacific Asia, and North America. Correlation matrices verified that label encoding introduced no spurious ordinal relationships. Numerical variables were deliberately maintained without scaling, as tree-based partition algorithms split on threshold values and remain invariant to monotonic feature transformations. Attempts to incorporate spatial shipping distances using geographic coordinates were abandoned because latitude and longitude coordinates for customers and transit nodes were absent from the dataset. Feature screening via mutual information metrics established shipping mode as the single strongest predictor of delivery outcomes with a score of 0.1854, followed by delivery duration at 0.0178, while interaction terms between shipping modes and geographic regions failed to surpass the cross-validation threshold required for model inclusion.

Model training and evaluation utilized Random Forest and XGBoost classification algorithms, selected for their demonstrated facility with mixed tabular data types, nonlinear feature relationships, and resilience against class imbalance without requiring data scaling. Validation protocols prioritized chronological integrity to prevent temporal data leakage across sequential periods. The 12,144 records were chronologically divided into a training partition of 70 percent containing 8501 records, a validation partition of 15 percent containing 1822 records, and a held-out test partition of 15 percent containing 1821 records. Hyperparameter tuning was conducted exclusively within the training set via fivefold stratified cross-validation. Optimal parameters for Random Forest established 200 estimators, a maximum tree depth of 12, a minimum split size of 5, and a minimum leaf size of 2, whereas XGBoost converged on 200 estimators, a maximum depth of 8, a learning rate of 0.05, and a subsample ratio of 0.8. Rolling time-series cross-validation confirmed stable temporal accuracy, yielding 0.612 for Random Forest and 0.598 for XGBoost across successive time windows.

On the held-out test partition, the Random Forest model achieved an overall accuracy of 63.5 percent and a weighted F1-score of 0.592, while the XGBoost classifier achieved an overall accuracy of 62.0 percent and a weighted F1-score of 0.590. To properly calibrate these metrics, performance was compared against a naive majority-class baseline that simply predicts delayed delivery for every transaction. Given that delayed shipments accounted for 57.9 percent of the test sample, the naive baseline achieved 57.9 percent accuracy. Consequently, Random Forest provided an absolute improvement of 5.6 percentage points over trivial guessing, and XGBoost yielded a gain of 4.1 percentage points. Macro-averaged receiver operating characteristic area under the curve reached 0.71 for Random Forest and 0.70 for XGBoost, with weighted curves scoring 0.74 and 0.73 respectively. These results demonstrated measurable but modest classification capabilities, positioning the models strictly as preliminary delay-screening tools rather than autonomous dispatching solutions.

Detailed class-level performance metrics illuminated substantial diagnostic disparities across the three delivery outcomes. Both algorithms proved most capable when identifying delayed deliveries, with Random Forest attaining a precision of 0.73 and recall of 0.76, while XGBoost reached a precision of 0.71 and recall of 0.77. In stark contrast, both classifiers struggled severely to identify on-time deliveries, where Random Forest produced a precision of 0.48 and an extremely low recall of 0.06, while XGBoost yielded a precision of 0.42 and a recall of 0.10. Density distribution analyses showed that early, on-time, and delayed deliveries clustered heavily around zero days, with overlapping duration signatures that prevented decision trees from discovering clean separation boundaries for on-time consignments.

Descriptive and statistical evaluations of operational features provided additional depth regarding transit factors and service tiers. Analysis of variance across the cleaned dataset confirmed statistically significant differences in delivery duration among shipping modes, yielding an F-statistic of 150.14 with significance below 0.001. Post-hoc testing through Tukey honestly significant difference tests indicated a significant difference of 10.60 days between Same Day and Standard Class shipping. Counterintuitively, First Class shipping exhibited the highest overall delay rate at 98.45 percent despite maintaining a relatively short mean duration of 27.52 days. Standard Class exhibited the lowest delay rate at 38.58 percent while recording the longest mean delivery duration of 47.29 days, and Same Day shipping averaged 13.14 days with a 56.55 percent delay frequency. Rather than indicating that faster shipping modes cause operational failure, the researcher noted that First Class selections likely reflect underlying customer urgency, difficult handling conditions, or high-risk transit requirements, meaning service level acts as a proxy for inherent order difficulty.

Geographical comparisons revealed that overall delay frequencies remained remarkably stable across worldwide market territories, hovering tightly within a narrow range between 57.08 percent in North America and 58.59 percent in Latin America. Conversely, mean delivery durations and transit variability diverged substantially between territories, with Africa recording the longest average delivery duration of 49.87 days alongside the highest variance, while Latin America maintained the shortest mean transit time of 39.10 days, and Europe exhibited the highest early delivery rate of 23.57 percent. Shipping mode allocations proved uniform across all markets, with Standard Class consistently capturing between 57 and 61 percent of volume and Same Day representing between 4 and 5 percent.

Feature importance rankings confirmed the central associative role of shipping mode across multiple evaluation frameworks, including tree-based Gini impurity reduction, gradient-boosted split gain, and mutual information. However, the study emphasized that feature importance reflects empirical predictive association within the specific secondary dataset rather than substantive causal mechanisms or actionable intervention targets. Order size and monetary total displayed negligible predictive relevance, each exhibiting correlation coefficients of minus 0.01 with delivery duration, contradicting conventional assumptions that order scale governs transit speed. Furthermore, the author noted that an exploratory composite regional resilience index was deliberately discarded from the final analysis because sensitivity evaluations proved that middle-tier regional rankings fluctuated widely under alternative weighting criteria, making separate descriptive reporting of individual performance metrics significantly more robust and scientifically defensible.

In synthesizing the analytical findings, the investigation established several explicit methodological boundaries and directions for future inquiry. Because the logistics dataset was obtained from an unverified secondary Kaggle upload, findings cannot be interpreted as universal operational constants across commercial supply chains. The marginal improvements over baseline classifiers, combined with near-total inability to identify on-time deliveries, reinforce that current predictive architectures cannot replace comprehensive human dispatching oversight. Furthermore, the methodological overlap between derived delivery duration and temporal target labels underscores the urgent need for leakage-resistant feature engineering in predictive logistics. The author concluded that future research must prioritize validated real-world datasets with verified organizational provenance, incorporate external environmental indicators such as local weather patterns and real-time transit congestion, and employ rigorous causal designs before recommending operational changes to global shipping strategies.

Subject of Research: Predictive machine learning algorithms for supply chain delivery disruption detection

Article Title: AI-powered predictive analytics for enhancing supply chain resilience and delivery outcomes

Article References: Khan, M. R. I. (2026). AI-powered predictive analytics for enhancing supply chain resilience and delivery outcomes. Discover Analytics, 4(1), Article 13. https://doi.org/10.1007/s44257-026-00065-6

Image Credits: AI Generated

DOI: 10.1007/s44257-026-00065-6

Keywords: Supply Chain Management, Predictive Analytics, Machine Learning, Random Forest, XGBoost, Logistics, Delivery Performance, Operational Resilience, AI-powered, predictive, analytics, enhancing