Temporal & Actionability Audit of Delivery-Risk Models
Demonstrating why prediction accuracy alone does not guarantee operational actionability in industrial supply chain ML.
Overview & Research Motivation
In industrial supply chain management, predictive machine learning models are widely deployed to forecast delivery delays. However, standard cross-validation practices often mask severe temporal data leakage and fail to verify whether predicted risk factors can be actionably intervened upon by human planners.
I conducted a rigorous empirical audit across 43,396 industrial order lines from an enterprise manufacturing dataset. Using CatBoost gradient boosting with strict forward time-based splits, probability calibration, clustered bootstrapping, and Individual Conditional Expectation (ICE) curves, I evaluated the transportability and operational actionability of delivery risk models across product lines.
The Core Systems Problem
- Random k-fold cross-validation in time-series supply chains creates artificial lookahead bias, producing overly optimistic accuracy metrics that collapse in production.
- High model feature importance does not imply that changing that feature (e.g. order quantity) will reduce delay risk, especially when products lack sufficient empirical support across quantity bands.
Why I Worked on It
"I wanted to expose the dangerous gap between statistical ML benchmarks and real shop-floor decision-making in industrial production operations."
My Specific Technical Contributions
- Processed and cleansed 43,396 industrial order transactions with multi-echelon supply chain parameters.
- Implemented forward temporal splits preventing temporal data leakage.
- Conducted Individual Conditional Expectation (ICE) and partial dependence analysis across product quantity bands.
- Proved that only 5 of 35 product families had sufficient empirical support for safe quantity-based operational intervention.
System Architecture & Verification Pipeline
Forward Temporal Splitting Pipeline
Partitions training, validation, and test datasets chronologically to simulate true real-time operational deployment.
Calibrated CatBoost Estimator
Optimized gradient boosted decision trees with isotonic probability calibration and clustered bootstrap confidence intervals.
Actionability & Support Verification
Computes ICE curves filtered by empirical density support to prevent out-of-distribution interventions.
Evaluation & Benchmark Results
Technical Stack & Tools
Known Limitations
- Evaluation conducted on discrete manufacturing supply chain logs; continuous process manufacturing may exhibit different lead-time dynamics.
What I Would Test Next
- Expanding actionability frameworks to include causal inference and counterfactual fairness constraints.
Connected Systems & Inquiries
Connecting component-level lifecycle emissions with persistent product state for circular electronics refurbishment.
A permissioned Cosmos SDK ledger for connecting supply chain state with MES and digital twin telemetry.
A sensor prototype and designed analytics pipeline for condition monitoring of industrial motors and pumps.