Building Scalable End-to-End Latency Metrics from Distributed Trace by Kusha Maharshi

In our sprawling microservices architecture at Bloomberg, timing requests from point A to point Z means navigating an alphabet's worth of services in between. With more than 50 billion spans a day, we process distributed trace into directed acyclic graphs (DAGs) with deep fan-outs and fan-ins that present concurrency chokepoints at scale. In this talk, we'll show how we wrestled complicated DAGs into shape, thus generating end-to-end latency metrics that drive SLOs, fire off alerts, and keep us all honest when latency starts creeping in!

1. A ScyllaDB Community
Building Scalable End-to-End
Latency Metrics From
Distributed Trace
Kusha Maharshi
Senior Software Engineer

2. Kusha Maharshi (she/her)

Senior Software Engineer at Bloomberg
■Telemetry infrastructure, distributed tracing
■My thought on p99s:
●Fast lane for the p99s, scenic route for the last 1%
■Distributed, streaming graph processing at scale
■Outside of work - dance, WNBA, reading

3. Storytime

■Recurring request: centralized end-to-end latency metrics
●In finance: milliseconds of delay = millions of dollars lost!
●State of things before: custom solutions by different engineering teams
■Distributed trace saves the day!
●Complex data:
■Long, fan-outs/fan-ins, >50B (& counting) daily spans
●So, we built a solution to handle that!

4. Distributed Trace
In The Wild

5. From equity order execution to viewing status of orders

6. Ensuring User Satisfaction

7. Why End-to-End Latency Monitoring Matters

■Monitor end-to-end client-facing workflows
●Client = external user, internal engineer team
■Latency spikes
●Something may be wonky!
●Alert engineers to debug
■Design & development
●End-to-end, A/B testing
■SLOs
●Also for managers, executives

8. Zooming In

9. Example Trace For Workflow






0s 2.3μs 5.7μs
consume
process
publish
publish
traceId = 0123456789abcdef





0s 2.4μs 5.8μs
consume
traceId = 0123456789abcde1





0s 2.1μs 5.9μs
consume
traceId = 0123456789abcde2
traceId = 0123456789abcde1
traceId = 0123456789abcde2
1
Consume queued messages
2
Batch and process
3
Publish to clients

10. Distributed Trace as a Directed Acyclic Graph (DAG)



t1: traceId
s11: spanId
consume: operation

Fan-in

11. End-to-End "Rule"

12. Rule: start = consume, end = publish
Get me the latency of
requests going from
consume to publish
Rule = consumeToPublish

13. Crux: Depth-First Search (DFS) through DAG
Difference between
start of start event
&
end of end event

Path 1: 3μs
Path 2: 3.5μs

14. Distributed Trace At Scale

15. Simple Traces At Scale

16. Enter "Fan-Ins"

17. Traces With Fan-ins (At Scale?)
Problem
Scaling with fan-ins

Solution #1
Put all spans from a
group of fanned-in traces
(aka trace bundle)
in the same partition

18. Solution #1

19. Aggregate Trace Bundles By Rule

20. Bundler: Merge Trace Bundles, Partition By Rule

21. Bundler: In-memory maps of potential trace-rule matches for trace bundles

22. Result: Aggregate Trace Bundles By Rule

23. Solution #1
■Ensures all spans from a trace bundle are in the same partition
■Works well when rules match a small set of spans & traces
■Scalability constraints
●Load imbalance
■Rule with high volume trace matches bloat dagger memory
■e.g., rules on middleware spans
●Fan-in memory bloat
■Fan-in traces that don't match any rules can exacerbate load imbalance

24. Solution #2

25. Lazy Approach: Send partial DFS info to instance with fan-in trace

26. Partial DFS Gossip

27. Solution #2
Instead of building trace bundles beforehand (greedy), this solution sends
out info about partial DFS paths when fan-in parents are encountered (lazy)
■Addresses load imbalance of solution #1
■Constraints
●Deep, chained fan-ins lead to long communication delays

28. Conclusion
End-to-end latency metrics from complex traces can be generated at scale
■To pick the best approach for you
●Employ operational knowledge about your trace data
●Mix and match
■End-to-end latency monitoring is crucial
●Trace provides a unified solution
●Client impact can be directly monitored
■SLOs
●Design and development of complex pipelines
■Distributed, streaming DAGs are hard and fun!

29. Thank you! Let’s connect.
Kusha Maharshi
[email protected]