Feature Store Evolution Under Cost Constraints: When Cost is Part of the Architecture

ShareChat's scaling ML Feature Store to handle 1B features/sec was just the start. Next challenge: cutting costs while keeping quality. Join Ivan & David to explore cloud cost optimization, Kubernetes waste reduction, and autoscaling Apache Flink. Perfect for #ML & #CloudDev. #P99Conf

1. A ScyllaDB Community
Feature Store Evolution Under Cost
Constraints: When Cost is Part of
the Architecture
David Malinge
Sr. Staff Software Engineer
ShareChat
Ivan Burmistrov
Principal Software Engineer

2. Previously on P99 Conf…
■ScyllaDB ftw
■Smart data model
■Various optimizations
■Cache locality
■…

3. Leadership:
“Great system! Can you
now build the one that
does the same, but 10x
cheaper? Thanks!”
Our reaction:

4. Cloud wants to rob you is complicated
■Billing is confusing: both AWS and GCP have ~40K SKUs
■Providers are not motivated to make it easy to understand and debug
■Easy to lose track of cost components


Step 1: ruthless cost inventory / attribution

5. Cost savings funnel

6. Cost savings funnel: Cloud traps

7. k8s wastage

8. Cloud traps: k8s wastage

9. Cloud traps: k8s wastage

10. Cloud traps: k8s wastage

11. Solution: Advanced k8s scheduler
■Open Source:



■Cloud specific:




■Commercial / Multi Cloud

12. “Cloud Tax”

13. Cloud traps: cross-AZ network egress (NIZE) cost
■3 zones
■Replication Factor = 3
(data gets copied to
all zones)

14. Naive reads: no settings in the client driver

15. Smarter reads: use token-aware routing

16. Smartest reads: + zone-aware routing

17. Smartest writes: everything we learned

18. Smartest+ writes: 2 zones

19. Smartest++ writes: 2 datacenters

20. Smartest+++ writes: a different cloud

21. Cost savings funnel: Compute optimization

22. ShareChat’s Feature Store
Case Study

23. Feature store: Architecture
More on compute: The Harsh Reality of Building a Realtime ML Feature Store (slides), QCon 2024

24. Feature Store

25. #1 Database scalability
■ScyllaDB clusters
■Flat-ish cost, no autoscaling
■Why flat-ish? Not autoscalable != not scalable
●Don’t scale for yearly peak!
●Kudos to Scylla support ??????
■True autoscaling incoming?
●Tablets in ScyllaDB 6.0

26. #2 Different workloads
■Read/Write pattern and how they affect each other

LIST ALL SERVICE_LEVELS;
service_level | timeout | workload_type | shares
---------------+---------+---------------+--------
serving | null | null | 1000
computing | null | null | 100
manual | null | null | 50
Read latency over time

27. #2 Different workloads

■Solutions
●Overscale Scylla cluster $$$
●Dual-datacenter - the classic advice: “isolate read and writes”
●usually leads to underutilized resources $$
●What if we could choose the latency level we want for reads vs writes?
■Scylla Workload prioritization $

28. #2 Different workloads

■Scylla only: use Workload Prioritization (WP)
●WP is great when different workloads are clear and consistent


⇒



■Other example: compaction strategy, see Scaling ShareChat’s ML Feature Store
LIST ALL SERVICE_LEVELS;
service_level | timeout | workload_type | shares
---------------+---------+---------------+--------
serving | null | null | 1000
computing | null | null | 100
manual | null | null | 50
⇒

29. Feature Serving Layer

30. Feature store costs: serving features
■Distributed GRPC service deployed on K8s
■Vast majority of the cost in compute
●Note: network costs can also be a problem
■For compute heavy K8s deployments, once you are past the the obvious pod
autoscaling (HPA/VPA), time to look into optimizations.
■Don’t optimize blindly: SETUP CONTINUOUS PROFILING!

31. Feature store costs: serving features
■The usual suspect: ser/de
■95%+ of our requests are cached
■Hottest path: request multiple entities => cache hits => merge => return
●aka: read from cache, deserialize, merge, serialize

Naive solution

32. Feature store costs: serving features
■The usual suspect: ser/de
■95%+ of our requests are cached
■Hottest path: request multiple entities => cache hits => merge => return
●aka: read from cache, deserialize, merge, serialize
■Biggest Win: partial deserialization!

33. Proto serde trick: encoding primer
■A message is series of key-value pairs (“record”)
●the key is the field number
●the value is encoded depending on type ⇒
●repeated fields emit one record per element
●maps are a repeated field under the hood
map<T, U> ⇔repeated entry<T,U>

{
“name” : “hello”,
“inner”: [{“val”: 1}, {“val”: 2}]
}

// protoscope output
3: {7: 1} // inner: {val: 1}
3: {7: 2} // inner: {val: 2}
source: https://protobuf.dev/programming-guides/encoding/#structure

34. Proto serde trick: encoding primer
message InnerMessage {
int32 val = 7;
}

message OuterMessage {
string name = 1;
repeated InnerMessage inner = 2;
}




// Example OuterMessage (as JSON)
{
“name” : “hello”,
“inner”: [{“val”: 1}, {“val”: 2}]
}


// protoscope output, 3 records:
1: {"hello"} // name: “hello”
2: {7: 1} // inner: {val: 1}
2: {7: 2} // inner: {val: 2}

35. Proto serde trick: how?
■Trick enabler:
●embedded messages and bytes
representation are the same (len-delimited)
■Why is this good?
●both are interchangeable!
●bytes deserialization is just a copy
■Ok.. but what do I do with bytes..?
●operations on maps and repeated fields!
●append, delete, swap, insert..
●All done without deserializing elements!
source: https://protobuf.dev/programming-guides/encoding/#structure

36. Proto serde trick: back to example
message InnerMessage {
int32 val = 8;
}

message OuterMessage {
string name = 1;
repeated InnerMessage inner = 2;
}






// OuterMessage can also
// be deserialized as:
message LazyOuterMessage {
bytes name = 1;
repeated bytes inner = 2;
}

37. Proto serde trick: back to cache

■Our cache stores serialized protos Features for a given entity
■Our response is basically a collection of Features for each entity requested

38. Proto serde trick: simple bench
■Repo: https://github.com/david-sharechat/lazy-proto
■Benchmarking “naive” merge vs lazy merge
●Appending 2 serialized protos with map and repeated field

39. Proto serde trick: simple bench

│ Naive │ Lazy |
sec/op | 16.821m ± 3% | 2.677m ± 6% -84.09% (p=0.000 n=10)

B/op | 8.479Mi ± 0% | 5.993Mi ± 0% -29.31% (p=0.000 n=10)

allocs/op | 141.16k ± 0% | 40.08k ± 0% -71.61% (p=0.000 n=10)

●6x faster with ⅓ of allocs!

40. Conclusion

41. Learnings

42. Before we part…
■We could not cover everything, please reach out if interested in these topics
●Computing windowed counter features (slashed costs by 5x)
●Flink autoscaling & state recovery (2x cost lever)
●Specialized Golang cache library (will likely open-source soon)
●… anything cheap and performant ;)

43. A ScyllaDB Community
Thanks for watching!
ShareChat

44. A ScyllaDB Community
Thanks for watching!
ShareChat

45. Feature Compute

46. Feature store costs: computing features
■Multiple Apache Flink jobs
■Problem: Flat cost, scaled for peak
■Obvious solution: autoscaling



Before CPU util chart
placeholder
After CPU util chart
placeholder
More on autoscaling Flink jobs: The Harsh Reality of Building a Realtime ML Feature Store (slides), QCon 2024