Optimizing the delivery promise with machine learning
KickstartAI x bol
KickstartAI joined forces with bol and engineers from five other partner companies to investigate if advanced machine learning could help make delivery promises more accurate. Ultimately, this project sought to uncover data-driven insights to increase platform revenue and improve customer satisfaction.
The Problem
Delivery speed matters when customers decide what to buy on bol. But sellers also need to avoid promising a delivery date they cannot meet. As a result, many partners build in extra time – and a large share of parcels arrive earlier than promised. This can negatively impact customer satisfaction and bol’s revenue alike.
Goal
The goal of the project is to develop a solution that predicts the optimal, shortest, and most reliable delivery promises for each product offer at any given order moment.
Approach
The work developed in three stages, together with bol’s engineering teams:
Build a baseline. Together with bol’s AI team, the project first established a benchmark model to improve on.
Put it to the test. Sixteen engineers from six companies spent two weeks trying different approaches to beat that baseline while meeting strict reliability requirements.
Look beyond the model. The team then worked with bol’s partner-facing teams to understand how the insights could be used in practice and what could be shown back to sellers.
Tech setup
Key system features include:
A wide range of signals : The models used factors such as order time, weekends and holidays, a seller’s past delivery performance, sales volumes, and product characteristics including size, weight, category, and brand.
Different ways of estimating risk. The team tested LightGBM and CatBoost models at different quantiles to find the point where a faster promise starts becoming too risky.
Advanced Clustering: Across the different models, historical delivery performance and the exact time an order was placed were among the strongest predictors.
Predictive Drivers: Across the different models, historical delivery performance and the exact time an order was placed were among the strongest predictors.
Conclusions
The most useful outcome was not a model ready for deployment, but a clearer understanding of the limits of prediction and where to focus next.
Determined Feasibility: We concluded it is not feasible to create a prediction model for this purpose, as accuracy levels were ultimately insufficient.
Strategic Shift: Uncovered that closer, direct collaboration with delivery partners is required to help them improve their own delivery promise settings.
Actionable Insights: Helped business and engineering teams deeply understand the underlying mechanics to begin experimenting with partner-facing dashboards.