“Enabling Smart Retail with Visual AI,” a Presentation from 365 Retail Markets

For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/09/enabling-smart-retail-with-visual-ai-a-presentation-from-365-retail-markets/

Himanshu Vajaria, Engineering Manager at 365 Retail Markets, presents the “Enabling Smart Retail with Visual AI” tutorial at the May 2024 Embedded Vision Summit.

Automated checkout systems are on the rise—preferred by customers and businesses alike. However, most systems rely on the customer scanning one product at a time and identify the product using a barcode. This adds friction to the process. The Stockwell cabinet from 365 Retail Markets eliminates this friction by using visual AI to recognize items from a catalog.

In this talk, Vajaria explains how the system works, the design decisions his company faced, and the operational and technical challenges and opportunities in the “Visual AI for Retail” space. After a brief overview of the business space, he delves into key technical aspects, including choice of camera and illumination, compute at the edge vs. the cloud, the AI inference pipeline and the data-labeling infrastructure for onboarding new products. He concludes with a look at collaboration opportunities in hardware, AI and data management.

1. Enabling Smart Retail with
Visual AI
Himanshu Vajaria
Research Scientist, Engineering Manager
365 Retail Markets

2. What is Unattended (Smart) Retail?
Customer scans
barcodes one at
a time and pays
Customer unlocks kiosk
with credit-card, takes
products and walks away
Micro-Market Self check outCV-AI smart machines
Self-Service Kiosks
AI Micro-Market
Cashier-less StoreVending Machine
Customer scans
products one at a
time, then pay using
POS terminal in the
market
Customer pre-
authenticates to
enter the store, take
products and leave
365 Retail Markets © 2024 2

3. Why is Unattended Retail Taking off?
30% of retail jobs are unfilled
-U.S. Chamber of Commerce, June
2023
66% prefer self checkout
14% will wait for SCO!
Health, hygiene, and
beauty products
US Retail Workers Are Fed Up
and Quitting at Record Rates
-Bloomberg News Sep 8, 2023
Manufacturing
Warehouses
Gig-economy
Labor Shortage 24/7 Anywhere Varied Products Customer Preference
365 Retail Markets © 2024 3

4. CV + AI + Retail
Product Identification
Spills Queue lengthsInventory Foot Traffic Fraud
365 Retail Markets © 2024 4

5. Empowering Global Retail
By Creating The Most
Innovative, Market-
leading Technologies
CEO and Founder
Joe Hessling
500+ Employees
Over 50,000 Points-of-Sale
worldwide
Operations in 30 Countries –and
growing!
Award Winning Company with
Great Products and Great People!
365 Retail Markets
365 Retail Markets © 2024
Largest Unattended Self-Checkout
Technology Provider in the world
5

6. PicoCooler/
PicoCoolerBreeze
MM6 Mini
PicoMarket
PicoMarket+
V5 Kiosk
V5 Kiosk with
Cash Accepter
MM6 Market
Kiosk
Stockwell 2.0
PicoVend
Mini
PicoVend
365 Beacon 365Pay App
365 Retail Markets © 2024
Product Portfolio
6

7. Stockwell 2.0
365 Retail Markets © 2024
https://www.youtube.com/watch?v=-uPv4IA24pE
7

8. Transaction Flow
AI RecognitionAuthenticate Record Transaction Receipt
Credit card
Digital wallet
App
Edge
Cloud
Hybrid
Auto-charge
Manual review
365 Retail Markets © 2024
Multiple cameras
Save video on device
8

9. AI Inference Pipeline
365 Retail Markets © 2024
Multiple
camera feeds
Object
detection per
camera frame
Aggregating ‘object crops’
from multiple cameras
Classification
model
Identify item
Why not use a single-shot detector?
9

10. •Inline labeling for transactions
•Low confidence
•Not enough training samples
•Manually reviewed transaction
•Use receipt items as a guide
•Use detection boxes
•Relabel items
•Use statistics for retraining
AI Labeling Pipeline
10365 Retail Markets © 2024

11. Self Driving vs Retail Object Recognition
Latency is critical Some delay acceptable
Generous sensor, compute budget Very cost conscious
No second chances Fallback is to manually review
Objects of interest do not vary Ever expanding product catalog
Product RecognitionSelf Driving
365 Retail Markets © 2024 11

12. Challenges in Object Recognition
12oz 16oz
Motion Blur Color Distortion Occlusions
Similar Packaging Small Sizes Seasonal Packaging
365 Retail Markets © 2024 12

13. Make it Harder! —Do it in Real Time
Confidence for new customers
Easily find item prices
Promotions and subsidies
Why real-time?
Technical challenges
Streaming high-resolution, high frame-rate videos from multiple cameras to the PC
CPU cycles to handle kiosk functions while recording/processing video-streams
Model management
Why edge processing?
Real-time identification
Cellular bandwidth cost/GB
Smarter use of compute
resources
365 Retail Markets © 2024 13

14. Only Cameras
or
Additional Sensors?
-Weight
-Time of Flight
Placement
Number
Field of View
Frame-rate
Resolution
Training Data
Partition the Catalog
Deployment
Update Strategy
Cost
CPU
GPU / TPU / ?
#USB ports, PCIe
ComputeSensing Cameras AI
14365 Retail Markets © 2024
AI-CV Kiosk Design

15. Camera Considerations
?
Camera Placement
FOV Resolution
Imaging Parameters
Frame-rate, resolution
Global / rolling shutter
#pixels on target
Compression
through the video
and AI pipeline
Gain, color correction
Anti-fog lenses
365 Retail Markets © 2024 15

16. AI Considerations —Compute
Cloud
-Device only uploads video
-Easy to scale
-Longer receipt time
-Bandwdith
Edge Hybrid
-Instant feedback
-Lower operational cost
-High upfront cost
-Hardware selection is critical
-Tuning software
-Model management
-Inflexible
-Store only uploads ‘clips’
-AI processes run on cloud
-Best of both worlds
-No live feedback
365 Retail Markets © 2024 16

17. AI Considerations —Model
Model UpdateModel Training Model Deployment
Partition catalog
Data collection
-Alpha stores
-Augmentation
-Label quality
Model selection
-Model capacity
-Inference time
Why update?
-Add new products
-Improve performance on
existing products
Metrics
-Quantify improvement
-Tradeoff performance on
existing vs new items
Trivial for the cloud
-One place to update
-Easy to monitor perf.
Edge considerations
-Cost to deploy
365 Retail Markets © 2024 17

18. Enablers
Custom Industrial Computers Data Labeling Platforms AI Platforms
Managed labeling platforms
•Data hosting,
•Labeling tools
•Managing labeling teams, QA
or
Offshore contractors
•Provide your own platform
•Train and manage workforce
•QA
Work with integrators to
customize every aspect of the PC
•USB ports
•Image
•GPU
•CPU
Complete solution for data
labeling, augmentation, model
training, optimization,
versioning, …
365 Retail Markets © 2024 18

19. Opportunities
High speed
Moderate resolution
Global shutter
Firmware control
Fog resistant lenses
Time-of-flight
Weight
What else is out there?
Reasonably priced edge
compute —per camera or
multi-camera
Existing pipeline of cameras
and edge processing
Sensors ComputeCameras and Lenses
365 Retail Markets © 2024 19

20. Generative AI for Training Data
365 Retail Markets © 2024
•Transactions with “unknown” items have to be manually reviewed
•Takes multiple transactions to collect the data, followed by training and deployment
•Would like recognize a new item on the very first transaction
•However, training the model requires data from live transactions
•Question?
•Given a stock image of an item, can we generate data of people’s hand holding and
moving the item?
20

21. Conclusions
Attended retail
Restaurants
Classrooms and schools
Machine vision
Vending machines → Smart kiosks
Sleek machines → New venues
Small format stores at transit
locations
Stores within stores
Cameras
Compute
Video pipelines
AI pipelines
Retail product
identification as a service
365 Retail Markets © 2024
Unattended retail is rapidly expanding
Overlap with other markets
Opportunities
21

22. Questions?
22365 Retail Markets © 2024

23. •Company websites
•365retailmarkets
•Stockwell
•About us
•Facilities
•What we do
Resources
23
•Product Spotlights
•PicoCooler
•PicoCooler Breeze
•PicoMarket
•Stockwell 2.0
•MM6 –MicroMarket Kiosk
•Order Ahead for Cafeteria Dining
365 Retail Markets © 2024