stackconf 2024 | Bring Your Chaos to Work Day by Dionysios Tsoumas

Join us for an emulated Chaos at Work Day. We’ll take over our pre-production environment and run a series of chaos experiments to test the resilience of our systems. We’ll be using a combination of LinkerD, Chaos Mesh, and K6 fault injections to simulate real-world scenarios and see how our monitoring systems respond. We want to be confident in answering some of the following questions:

How does our monitoring and alerting stack (Grafana, Loki, Tempo + Thanos) behave under unexpected behaviors?
How do our systems respond to network latency and failures? How quickly will our on-call team be notified?
Does our observability stack provide the right level of detail to understand what’s happening?
Can we build any interesting dashboards to help us understand the impact of chaos on our systems?
Moreover, can we make it fun? We’ll be running a series of guess games throughout the talk to keep everyone engaged and entertained.

1. Bring your chaos
To work Day
Dionysis Tsoumas, Director of DevOps

2. Ok, let’s do this!
01
02
03
04
05
Our stack
How did we get here?
Our tools
Tests and dashboards
Gamifying results

3. Bonus
What is chaos
engineering?

4. Our stack
Monitoring & observability
●Prometheus operator
●Thanos
●Grafana
●Loki
●Tempo
●K6
●OpsGenie
The Rest
●Kubernetes clusters on GCP
●GCP monitoring for fallback
●Sentry
●LinkerD
●Chaos mesh

5. How did we get here?
Is chaos engineering still relevant?

6. Right-click image to replace
Login service
Almost real life example
Bundles service
Subscriptions
service
User request

7. 2 days event, where we tried
to bring down our staging
and monitoring platforms
Bring your chaos to work
Possibilities were endless

8. A few special words
about LinkerD
https://linkerd.io/

9. And of course chaos
mesh
https://chaos-mesh.org/

10. Prep work: Let’s talk about tests

Chaos mesh
●Stress Test
●Pod Failures
●Network Delay
LinkerD
●Fault injection

11. Should we have a peak?
(Of course we should)

12. Prep work: Let’s talk about dashboards

13. Our roles
●Chronicler – Keeping notes
●Responder – Keeping track of alerts
●(2x) Maestros – Running the experiments
●Firefighter – Handling communication
●Watcher – Monitoring for errors

14. Is it worth emulating users?
(Yes and no)
Quick question:

15. Ok, time for results and
interesting pieces for next
steps.

What did we learn?

16. Right-click image to replace
Let’s play a game.

17. Right-click image to replace
How many P1 alerts we triggered?


Left: Between 0 - 5 Right: Between 6 - 15

18. We only got one
(1) P1 alert
-Good old Prometheus
(PrometheusIsntanceIsDown)






Action Items based on:
-P1 alerts are too flexible
-Or our tests were not aggressive
enough

19. Right-click image to replace
What about P2 vs P3?


Left: More P2 Right: More P3

20. We only got P3
alerts
- Triggered by LinkerD -
ProxyInboundSuccessRate
(You know what this means)






Action Items:
-At least for monitoring
namespaces, P3->P2/P1

21. Right-click image to replace
What about P2?


Left: We didn’t have
any P2s
Right: Eventual
consistency

22. Eventually we
would get P2/P3
alerts
- Triggered at 20 min
-NotEnoughReplicas or OOM






Action Items:
-Improve alerts on monitoring
namespaces
-P3 at 15’, P2 at 30’

23. Ou biggest finding?
A retrying mechanism would solve most of our problems and help a lot with
blank screens. Also, we are not as cool as we thought.

●We introduced LinkerD’s Retry fallback mechanism
●Retries, timeouts and retry budgets
●You can monitor them as well!

24. Thanks for
participating!
Have more questions?
Reach me at [email protected]
LinkedIn: Dionysis Tsoumas