What Flow Metrics Teaches Us About Designing Resilient Systems by Mourjo Sen

Flow metrics help distinguish valuable work ("goodput") from total work done ("throughput"). This talk explores how flow metrics improve agile teams and system resilience by tracking velocity, time, efficiency, and load—key to designing scalable systems that handle real-world demands efficiently.

1. A ScyllaDB Community
What Flow Metrics
Teaches Us About Designing
Resilient Systems
Mourjo Sen
Senior Software Engineer

2. About Me
■Senior Software Engineer at Booking.com
■Building backend systems for 10 years
■Writing about system resilience since 2020

mourjo.medium.com

3. Everything Fails. All the time.
– Werner Vogels

4. Metrics Should Help Solve Problems

5. Metrics Should Help Solve Problems

6. Metrics Should Help Solve Problems
Latency
increase
(impact)
github.com/mourjo/monster-scale-2025

7. Metrics Should Help Solve Problems
Increase in
the number
of requests
github.com/mourjo/monster-scale-2025

8. Metrics Should Help Solve Problems
Everything is
okay
(200 status
codes)
github.com/mourjo/monster-scale-2025

9. Metrics Should Help Solve Problems
No impact on
processing
time
github.com/mourjo/monster-scale-2025

10. Metrics Should Help Solve Problems
Disconnect between
metrics and value
proposition
github.com/mourjo/monster-scale-2025

11. Software: A Means to Deliver
Business Value

12. Building Software: An Analogy

13. Building Software: An Analogy

14. Building Software: An Analogy

15. Building Software: An Analogy
Tasks to do

16. Building Software: An Analogy
Teams work on
the tasks
Tasks to do

17. Building Software: An Analogy
Teams work on
the tasks
Tasks to do

18. Building Software: An Analogy
Teams work on
the tasks
Tasks to do
Assembled
product

19. Building Software: An Analogy
Teams work on
the tasks
Business Value
Tasks to do
Assembled
product

20. What is Flow Metrics?

21. ■The customer wants the final result
Business Value Does Not Magically Appear

22. ■The customer wants the final result, not individual work items
Business Value Does Not Magically Appear

23. ■The customer wants the final result, not individual work items
■Flow is the movement of work items from potential to concrete value

Business Value Does Not Magically Appear

24. ■The customer wants the final result, not individual work items
■Flow is the movement of work items from potential to concrete value
■Flow metrics measure flow

Business Value Does Not Magically Appear

25. Flow Metrics in Software Engineering
Software
Engineering

26. Flow Metrics in Software Engineering
WIP

Software
Engineering

27. Flow Metrics in Software Engineering
WIP
Age
Software
Engineering

28. Flow Metrics in Software Engineering
WIP
Age
Cycle Time

Software
Engineering

29. Flow Metrics in Software Engineering
WIP
Age
Cycle Time
Throughput
Software
Engineering

30. Flow Metrics in Systems
WIP
Age
Cycle Time
Throughput
RequestsUsers Servers

31. Applying Flow Metrics to
Software Systems

32. Flow Metrics Applied to Systems

33. Flow Metrics Applied to Systems

34. Flow Metrics Applied to Systems

35. Flow Metrics Applied to Systems

36. Flow Metrics Applied to Systems

37. Scenario 1: Spike in Incoming Requests

38. Scenario 1: Spike in Incoming Requests
Increase in WIP

39. Scenario 1: Spike in Incoming Requests
Increase in WIP

40. Scenario 1: Spike in Incoming Requests

41. Scenario 1: Spike in Incoming Requests

42. Scenario 1: Spike in Incoming Requests

43. Scenario 1: Spike in Incoming Requests
Increase in age

44. Scenario 1: Spike in Incoming Requests
Increase in age

45. Scenario 1: Spike in Incoming Requests
Increase in age

46. Scenario 1: Spike in Incoming Requests
Increase in age

47. Scenario 1: Spike in Incoming Requests
Increase in
cycle time

48. Scenario 2: Degraded Dependency

49. Scenario 2: Degraded Dependency

50. Scenario 2: Degraded Dependency

51. Scenario 2: Degraded Dependency

52. Scenario 2: Degraded Dependency
Increase in WIP

53. Scenario 2: Degraded Dependency
Increase in age
Increase in WIP

54. Scenario 2: Degraded Dependency
Increase in age
Increase in WIP
Decrease in
throughput
Increase in
Cycle Time

55. ■If there is congestion
■WIP increases
■Age increases
■Cycle time will increase
Flow Metrics Applied to Systems

56. ■If there is congestion
■WIP increases
■Age increases
■Cycle time will increase
Flow Metrics Applied to Systems
Leading indicators

57. Detecting Congestion with Flow Metrics

58. Detecting Congestion with Flow Metrics

59. Detecting Congestion with Flow Metrics

60. Detecting Congestion with Flow Metrics

61. Detecting Congestion with Flow Metrics

62. Detecting Congestion with Flow Metrics
Impact observed
by the client

63. Detecting Congestion with Flow Metrics
Leading
Indicators
Impact observed
by the client

64. Detecting Congestion with Flow Metrics
Leading
Indicators
Impact observed
by the client

65. Detecting Congestion with Flow Metrics
Leading
Indicators
Requests spend
most of their
age just waiting
Impact observed
by the client

66. Detecting Congestion with Flow Metrics
Leading
Indicators
There are
too many
requests in
the system
Impact observed
by the client

67. Detecting Congestion with Flow Metrics
Source

68. Detecting Congestion with Flow Metrics
Source
Just detecting
congestion is not
good enough

69. On-the-fly Resilience with
Flow Metrics

70. Beyond Monitoring: Limiting WIP and Age
Server Threads

71. Queue length slows down
flow of business value
Beyond Monitoring: Limiting WIP and Age
Server Threads

72. Server Threads
Worker ThreadsLimited
Queue Size
Beyond Monitoring: Limiting WIP and Age

73. Limited
Queue Size
Age filter
Beyond Monitoring: Limiting WIP and Age
Server Threads
Worker Threads

74. Gatekeeper
Threads
Limited
Queue Size
Age filter
Beyond Monitoring: Limiting WIP and Age
Server Threads
Worker Threads

75. Gatekeeper
Threads
Limited
Queue Size
Age filter
Beyond Monitoring: Limiting WIP and Age
Server Threads
Worker Threads

76. Why Limit WIP and Age?
■Bounded wait-time for clients
Gatekeeper
Threads
Limited
Queue Size
Age filter

77. Why Limit WIP and Age?
■Bounded wait-time for clients
■Effective communication when there is congestion
Gatekeeper
Threads
Limited
Queue Size
Age filter

78. Why Limit WIP and Age?
■Bounded wait-time for clients
■Effective communication when there is congestion
■Faster recovery after incident
Gatekeeper
Threads
Limited
Queue Size
Age filter

79. Result 1: Spike in Incoming Requests
github.com/mourjo/monster-scale-2025
Default Spring Boot Server

80. Result 1: Spike in Incoming Requests
github.com/mourjo/monster-scale-2025
Default Spring Boot Server
WIP-Age Limiting Server

81. Result 1: Spike in Incoming Requests
Responds in timeUnresponsive
github.com/mourjo/monster-scale-2025
Default Spring Boot Server
WIP-Age Limiting Server

82. Result 1: Spike in Incoming Requests
Recovers immediately

Time to recover is longer

github.com/mourjo/monster-scale-2025
Default Spring Boot Server
WIP-Age Limiting Server

83. Result 1: Spike in Incoming Requests
Graceful degradationPretends everything is 200 OK
github.com/mourjo/monster-scale-2025
Default Spring Boot Server
WIP-Age Limiting Server

84. Result 2: Degraded Dependency
github.com/mourjo/monster-scale-2025
Default Spring Boot Server
WIP-Age Limiting Server

85. Result 2: Degraded Dependency
Responds in timeUnresponsive
github.com/mourjo/monster-scale-2025
Default Spring Boot Server
WIP-Age Limiting Server

86. Result 2: Degraded Dependency
Recovers immediately

Time to recover is longer

github.com/mourjo/monster-scale-2025
Default Spring Boot Server
WIP-Age Limiting Server

87. Result 2: Degraded Dependency
Graceful degradationPretends everything is 200 OK
github.com/mourjo/monster-scale-2025
Default Spring Boot Server
WIP-Age Limiting Server

88. End-to-End Resilience with
Flow Metrics

89. Local Measurements to Global Resilience

90. Local Measurements to Global Resilience

91. Local Measurements to Global Resilience

92. Local Measurements to Global Resilience

93. Local Measurements to Global Resilience

94. Local Measurements to Global Resilience

95. Local Measurements to Global Resilience
Measure the flow of
individual requests, not
the server’s capacity
?

96. Independent of System Characteristics
■Machines in a Cluster
■Memory and Garbage Collection
■Distribution of type of requests
■Number of downstream dependencies
■CPU Utilization
■…

97. Source

98. Source

99. Source

100. Source

101. Resilience By Design is an
Insurance Policy

102. Your Service Is at Risk
■Sudden burst of traffic
■Slow dependency
■Latency bugs
■…

103. Your Service Is at Risk
■Sudden burst of traffic
■Slow dependency
■Latency bugs
■…
Flow metrics detects and protects
your service against cascading
failures

104. Your Service System Is at Risk
■Sudden burst of traffic
■Slow dependency
■Latency bugs
■…
Flow metrics detects and protects
your system against cascading
failures

105. Your Service System Is at Risk
■Sudden burst of traffic
■Slow dependency
■Latency bugs
■…
Flow metrics detects and protects
your system against cascading
failures

106. An Insurance Policy: Resilience With Flow Metrics
■Acknowledgement of limits of a system

107. ■Acknowledgement of limits of a system
■Effective communication during outages

An Insurance Policy: Resilience With Flow Metrics

108. ■Acknowledgement of limits of a system
■Effective communication during outages
■Independent of system characteristics
An Insurance Policy: Resilience With Flow Metrics

109. An Insurance Policy: Resilience With Flow Metrics
■Acknowledgement of limits of a system
■Effective communication during outages
■Independent of system characteristics
■Simple to implement with little overhead

110. An Insurance Policy: Resilience With Flow Metrics
■Acknowledgement of limits of a system
■Effective communication during outages
■Independent of system characteristics
■Simple to implement with little overhead

Embed Flow of Business Value
into System Design
github.com/mourjo/monster-scale-2025

111. Thank you!
Questions?
[email protected]
linkedin.com/in/mourjo

https://mourjo.me
github.com/mourjo

What Flow Metrics
Teaches Us About Designing
Resilient Systems
github.com/mourjo/monster-scale-2025