Efficient Deduplication in ReSys at Scale: How We Managed to Reduce Server Cost by 90% by Andrei Manakov

Efficient deduplication is key to delivering relevant recommendations at scale. Andrei & Ashish will share how they optimized for 300M+ users, cutting costs by 90% while reducing latency. They cover how ScyllaDB, Apache Flink, Redpanda & Google Dataflow powered this evolution.

1. A ScyllaDB Community
Efficient Deduplication in
RecSys at Scale:
How ShareChat Reduced Server Costs by 90%
Andrei Manakov
Senior Staff Software Engineer

2. Andrei Manakov (he/him)
■More than 13 years experience in industry
■Write posts in the blog “Andrei and The
Optimization Stone” about distributed
systems
■Harry Potter Fan


Your photo
goes here,
smile :)

3. Presentation Agenda
■Sharechat Overview
■What Is A Deduplication In RecSys?
■Initial System State
■Optimisations
■Conclusions

4. What Is Deduplication In
RecSys?

7. Initial State

8. Concept

9. Components

10. Optimisations

11. Cost Efficient Technologies

12. Streaming jobs

13. Database

14. Don’t Waste Your Resources

15. Domain Driven Optimisation

16. Domain Driven Optimisation

17. Domain Driven Optimisation

18. Storage Size Optimization: why is it important?

19. Storage Size Optimization: Compression algorithms
■PostIDs are sequential numbers. What are our options?
■Bloom Filter
■Offers pretty nice compression
■Probabilistic in nature, returns false positives
■Impossible to parse postIDs back from the filter
■Roaring Bitmaps
■Offers good compression only when postIDs batch is bigger >300
■Delta Based Compression
■Better compression due to sequential nature of postIDs
■Deterministic and can decompress to get postIDs set

20. Optimized Compression Schema
This algorithm achieved 60% of storage reduction

Overall Cost: 30% -> 20%

21. Sent Post Optimization

22. Sent Post Optimization
- Redis per GB cost was higher than ScyllaDB
- Sent + Viewed posts in the same DB - easier
maintenance

Overall Cost: 20% -> 17%

23. Cloud Specific Cost Reduction
■Spot Nodes -> Cast AI
■Know your network pricing: Optimize traffic between
availability zones
■Redpanda -> Follower-Fetching
■ScyllaDB
■Token-aware policy
■Rack-aware policy
■K8s
■Topology aware routing
■Service mesh


Overall Cost: 17% -> 10%

24. Stay in Touch
Andrei Manakov
[email protected]
@AndreyManakov
https://andection.bsky.social/
https://www.linkedin.com/in/andrei-mana
kov-69228a81/
https://andection.substack.com/