You cant Test everything, but you should monitor it (OpenSearchCon)

We had an incident which did occur in our warehouse at KRUU. The downloading of the photos was very slow from one to the other day - well we thought that the problem started on this day.

Actually we did notice this very late and the problem started two years ago but we did notice this very late due to the reduced rentals because of Covid19.

This will never happen again thanks to our new metrics and alerting powered by the OpenSearch!

1. YOU CAN’T TEST EVERYTHING
BUT YOU SHOULD MONITOR IT!
OpenSearchCon Europe 2024
Our journey to OpenSearch

2. Hi, I am Michi!

Head of Code at
@michilehr

3. "As Europe's leading photo booth provider, we have
made it our mission to help our brides and grooms
with their complete journey to their dream wedding.
This is something we work tirelessly on with our
team."
What is doing?
Philipp Schreiber - Co-Founder, KRUU.com

6. Photo Booth
Cycle
Photo Booth
Cycle

8. The Incident

9. Photo Booth
Cycle

10. Good
~12 MB/s

11. Bad
0.95 MB/s

12. ??????

13. -Hardware error
-Bug in code
-Network
-OS
What could it be?

14. Transferring some sample
data was fast
What could it be?

15. 1.When did it start?
2.Why did it happen?
3.How to prevent?
4.How to notice early?
Investigate

16. When did it start?
We had data in our Slack Channel, but…

17. 1.Write a script to extract the data as CSV
2.Import data to MySQL
3.Write query to aggregate by day
4.Create nice chart

18. Many
hours
later

19. Started long time ago…

20. What happened?

21. First day at the new warehouse

22. Network configuration error

23. How to prevent or test?

24. How to notice early?

25. ??????

26. Metrics
??????

31. But HOW to notice early?

32. Monitoring
and
Alerts!

33. Monitor

34. Monitor Data Source

35. Monitor Data Source Query

36. Monitor Data Source Query Trigger

37. Monitor Data Source Query Trigger Notification

40. Online Photo Booths alert
What else?

41. 404 monitoring
What’s Next?

42. YOU CAN’T TEST EVERYTHING
BUT YOU SHOULD MONITOR IT!

43. Thank you for your time!

Questions?

Feedback?

Notes?