ML @ Instacart: Improving the quality of On-demand Grocery

Mitigate impact of stockouts in Grocery delivery through machine learning. Near real-time inventory prediction.

1. Sharath Rao
http://www.pivenworld.com/
ML@Instacart:
Improving Quality of
On-Demand Grocery

2. #Orders with “Face Masks” on Instacart

4. ●
●
●
@sharathrao

5. 01
02
03

6. Groceries
delivered
to your
doorstep
or ready
for pickup
from
stores you
love
in as little
as an hour
+ + + =

7. and many
more ….

8. v

9. Customers
Shoppers
Products
(Advertisers)
Search
Advertising
Shopping
Logistics
Customer Service
Inventory
Picking
Loyalty
Stores
(Retailers)

10. Select Store Build Cart
Delivery/Pickup
Time Selection

12. 01

13. ●
●
●
●
●
●
●
●
●
●
●
●
●
Long tail of infrequent occurrences with
occasional severe impact if not addressed

14. Stockouts Poor Replacement

15. Source: https://hbr.org/2004/05/stock-outs-cause-walkouts

16. ●
●
●
●

17. “Don’t show
me items that
can’t be
fulfilled”
“Call/chat during
picking regarding
replacements”
“Warn me about
stock level and
let me specify the
replacement”
“Trust the
shopper to
replace”
“Do not
replace item”
More permissive
Convenience Focus
More Proactive
Precision Focus

18. ●Item Found Rate
○% of ordered items that are fulfilled
●Replacement Satisfaction Rate
○% of replacements implicitly/explicitly approved by customer

19. 02

21. In-store activity assumed as ground truth of item being in the store
Model Probability(Found) as a supervised binary classification problem

22. Aggregate Found Rate Statistics at higher levels (Critical for Tail Items)
●Product
●Retailer + Product
●Retailer + Product + Region
XGBoost Model with ~100 features
Example Item Level Features (Effective for Popular Items)
●Time since last not-found/found
●Found Rate in last 6m, 3m, 1 week, 1 day, 6 hours

23. Availability scores low across US
Even in stores where it hasn’t been bought (due to
product level features)
A store in Mission District:
Availability score: bottom 2 percentile
Past found rate: extremely low
From the Vice News report: “The milk
substitute has become so popular in places
like Brooklyn that shortages have broken out,
and a grey market for the most sought-after
brand of the stuff — Oatly — has popped up
online. Sometimes liters of Oatly can sell for
as much as $18 each.”

24. What comes next?
You’ve been freed
Do you know
how hard it is to lead?
-
Source: https://smcl.org/blogs/post/wait-for-it/

25. 03

26. ●Retailer Location Selection
●Replacement Suggestions
during fulfillment/picking
●Picking Item List Ranking
●Search and Merchandising Ranking
●Low Stock Inform and Mitigate
●Post-Checkout Experience
●Chat with shopper during
fulfillment/picking
Customer Experience Shopper Experience

27. ●Eroding trust in Catalog
●Customer Churn
Show all relevant
items including
low stock
Remove all low
stock items
●Eroding trust in Fulfillment
●Lower shopper productivity
●Customer Churn
How can we create the optimal search experience?

28. Good improvement in Found Rate
Unacceptable drop in Search Conversion Rate

Hidden Items
Query: “Milk chocolate”
We were often removing items customers really wanted!

29. Demoted Items
No drop in Search Conversion Rate
Low improvement in Found Rate

Customers were ordering demoted items anyway!
Query: “Milk chocolate”

31. ●Use data collected from demotion experiment
●Learn a simple model/heuristic to predict whether a
demoted item would be ordered given search
Key Insight:
Highly relevant
Query: “Milk chocolate”

32. Top 2 features describing search context
●# of results
●# of demoted results from top 5
# of items demoted in top 5
results
Don’t operate here
Ensure that the search results are relevant, mostly in stock and retains good selection of products

34. “Don’t show
me items that
can’t be
fulfilled”
“Call/chat during
picking regarding
replacements”
“Warn me about
stock level and
let me specify the
replacement”
“Trust the
shopper to
replace”
“Do not
replace item”
More permissive
Convenience Focus
More Proactive
Precision Focus

35. Replacement suggestions are generated by an ML recommendation system

36. Do not replaceMore replacement
suggestions

37. ●Retailer Location Selection
●Replacement Suggestions
during fulfillment
●Picking Item List Ranking
●Search and Merchandising Ranking
●Low Stock Inform and Mitigate
●Post-Checkout Experience
●Chat with shopper during
fulfillment/picking
40% relative reduction
in low quality deliveries
Customer Experience Shopper Experience

38. ●Pick problems where ML can make unique contributions
○Hiring/training product minded ML Engineers is challenging
●Pick problems that are fundamental to the business with broad applications
○Instacart examples: Item Availability, Replacement recommendation, Product Targeting
●New use cases of existing models are often higher ROI than improving the model*
●Design and PM are key partners in getting the most value out of ML efforts