Metrics, logs, traces, and mayhem_ introducing an observability adventure game powered by Grafana Alloy and OTel

This hands-on workshop transforms learning about observability into an engaging adventure!

Step into a text-based game where you’ll master essential observability tools—metrics, logs, and traces—and discover how to leverage them effectively for real-world troubleshooting and gaining critical insights into your applications.

Powered by OpenTelemetry (OTel) and the Grafana stack, this session offers a unique, interactive environment for exploring the critical components of a robust observability strategy. You’ll gain practical experience identifying, understanding, and resolving complex system issues.

Whether you’re a developer, operator, or just curious about understanding your system’s health, you’ll leave with a solid foundation on how OpenTelemetry and core observability tools can empower you to solve complex problems confidently.

You’re also welcome to work in pairs and collaborate on your observability adventure!

Pre-requisites: Docker and Git are installed on your machine, or access to a web browser to run the workshop on a Killercoda playground

1. Logs, Metrics, Traces and Mayhem:
An observability adventure game powered by Grafana Alloy and OTel

2. Who am I?
Imma Valls
Staff Developer Advocate
Grafana Labs
https://eyeveebee.dev/
https://github.com/immavalls
https://www.linkedin.com/in/imma-valls

3. Setting the scene
1.What is observability?
2.What can go wrong?
3.Some telemetry signals: metrics, logs & traces
4.Let’s play!
5.Not just playing …
6.More observability adventures, please!
7.Takeaways

4. Black Box
Input Output
1.What is observability?
print(“Why is this broken?!” )

5. 2. Observability Lessons
Matchmaking DB
Failures
2021
Performance
Problems
2020
Login & Scaling
Issues
2023
When things go wrong…

6. 2. Why does this happen?
●Complexity
○Dawn of Microservices
●Companies are victim of their own success
○Demand
●We are all human…
○Bad code
○Bad package
○Users do User things

7. 3. Key components of observability
Metrics
“What”
Logs
“Why”
Traces
“Where”

8. 3.1. Metrics - What happened?
●Quantifiable data
●Time series
●Unique name
●Label key-value pairs
●Metric types (counter, gauge, etc)
http_requests_total{method="post",code="200",path="/api/v1/users"} 1234
node_memory_usage_bytes{instance="server-01",job="node-exporter"} 8432678912

9. 3.2. Logs - Why did it happen?
●Bread crumbs…
●Log entries can take many
forms.
[2024-04-09T12:44:24.076] WARN: One of your party members has fallen ill! We should find medicine soon.
[2024-04-09T13:44:24.076] ERROR: Your party member has died…
Log Entry
Timestamp
Severity
Message
Metadata

10. 3.3. Traces - Where did it happen?
●Track requests across services
●Traces are made of spans
●Help pinpoint slowdowns and
failures
●Essential for distributed systems

11. 3.4. Microservices Example
Customers’ interface for placing orders
Kitchen Service
Manage order preparation
Delivery System
Tracking and final delivery
Pizza Delivery Application

12. 3.5. Exemplars
●A rich data point that links a high-level
metric observation (e.g. a latency spike) to
a specific trace (span / trace ID).
●Bridges the gap between Metrics and
Traces.
●In Grafana, exemplars appear as markers
on your metric graphs / link .
https://grafana.com/docs/grafana/latest/fundamentals/exemplars

13. 3.4. Exemplars
●Pizzas are arriving late (metric), why?

14. 3.4. Exemplars
●Pizzas are arriving late (metric), why?
●From the latency graph, go to traces
associated with those delays, and check
where the delay happens (exemplar)

15. 3.4. Exemplars
●Pizzas are arriving late (metric), why?
●From the latency graph, go to traces
associated with those delays, and check
where the delay happens (exemplar)
●It’s raining where orders are delayed!

16. How do all these telemetry signals
come together?

17. 4. Let’s play!
●Forge an awesome sword
●Make it more powerful
●Take on a quest
●Defeat the wizard
●Save the town!
Teleprinter

18. 4.1. Creating the adventure!
●Key components:
○OpenTelemetry
○Grafana Alloy
○Prometheus
○Loki
○Tempo
○Grafana
main.py

19. 4.2. OpenTelemetry
●A method of standardizing telemetry signals:
○Framework
○API
○SDK
○Collector
●We use the Python SDK to standardize how we
write metrics and logs

main.py

20. 4.3. Grafana Alloy
●OpenTelemetry collector distribution
●Alloy performs two tasks in our adventure
○Receives
○Sends

21. 4.4. Telemetry Storage
Loki
●Logs Store
●LogQL
Prometheus
●Metrics Store
●PromQL

Tempo
●Traces Store
●TraceQL

22. 4.5. Grafana
●Open-source Observability Platform
●Visualise, Monitor, and Analyse
●Big tent philosophy
●Alerts

23. 4. Let’s play!
https://github.com/grafana/adventure

24. 4. Let’s play!
https://bit.ly/o11y-workshop-openfest-25

25. 4. Let’s play!
1.What metrics are useful to play the game?
2.What useful log appears when we accept the offer
from the mysterious man in the village?
3.What dashboard allows us to view traces?
4.What does an exemplar allow us to view?
5.How do we navigate from logs to traces?
6.How do we navigate telemetry signals without using
logQL or promQL?
7.Did we have any useful alerts set up?
8.Did anyone spot an Easter egg???????

26. 1.What metrics are useful to play the game?
4. Let’s play!

27. 2. What useful log appears when we accept the offer
from the mysterious man in the village?
4. Let’s play!

28. 3. What dashboard allows us to view traces?
4. Let’s play!

29. 4. What does an exemplar allow us to view?
4. Let’s play!

30. 5. How do we navigate from logs to traces?
4. Let’s play!

31. 4. Let’s play!
6. How do we navigate telemetry signals without using
logQL or promQL?

32. 6. How do we navigate telemetry signals without using
logQL or promQL?
https://grafana.com/blog/2025/02/20/grafana-drilldown-apps-the-improved-queryless-experience-formerly-known-as-the-explore-apps/

4. Let’s play!

33. 7. Did we have any useful alerts set up?
4. Let’s play!

34. 4. Let’s play!
7. Did we have any useful alerts set up?

35. 8. Did anyone spot an Easter egg???????
4. Let’s play!

36. 5. Not just playing …
1.Create a cheat counter




2.Modify the dashboard to add the counter
https://opentelemetry.io/docs/specs/otel/metrics/api

37. 5. Not just playing…
1.Create a cheat counter

38. 5. Not just playing …
2. Modify the dashboard to add the counter

39. 6. More observability adventures, please!
https://github.com/grafana/alloy-scenarios/tree/main/game-of-tracing
●OpenTelemetry tracing
fundamentals
●Context propagation & span links
●Correlation for debugging

40. 6. More observability adventures, please!
https://grafana.com/blog/2025/08/11/learn-opentelemetry-traci
ng-through-a-grand-strategy-game-introducing-game-of-traces/

41. https://killercoda.com/grafana-labs/course/workshops/game-of-traces
6. More observability adventures, please!

42. 6. More observability adventures, please!

43. 7. Takeaways
●Observability is no longer a luxury
●What, Why, Where
●OpenTelemetry unifies telemetry signals
●Learning Observability doesn’t have to be a chore

44. Thanks for listening, adventurer!
Have ye any questions?

45. Join our Community!