Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)

Because observability is such a broad topic – and often something we learn on the job – it can feel like there’s too much to learn at once. But you don’t have to tackle everything and can start with the basics and build from there!

Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.

No matter what tooling is in place, there are still observability fundamentals that developers should know. That’s why I’ve put together a primer on the different telemetry types, when to use them, how to understand the data journey, and what to look for in time series graphs.

1. @paigerduty
Observability
Concepts
EVERY
Developer
Should Know

2. @paigerduty 2
SHOULD

3. @paigerduty 3
NEW GRAD

4. @paigerduty 4
NEW GRAD SR. DEV

5. @paigerduty 5
NEW GRAD SR. DEV STAFF

6. @paigerduty

7. @paigerduty

8. @paigerduty

9. @paigerduty
O11y
Knowledge
Gaps
9

10. @paigerduty
O11y
Knowledge
Gaps
10
Tool Sprawl

11. @paigerduty
O11y
Knowledge
Gaps
11
Tool Sprawl
Hidden
Processing

12. @paigerduty
O11y
Knowledge
Gaps
12
Tool Sprawl
No / Shallow
Onboarding
Hidden
Processing

13. @paigerduty
Auto-magic
O11y
Knowledge
Gaps
13
Tool Sprawl
Hidden
Processing
No / Shallow
Onboarding

14. @paigerduty 14Image: smashingstocks on freepik

15. @paigerduty 15Image: smashingstocks on freepik

16. @paigerduty
•Telemetry
Types

•Your Telemetry
Journey

•Interpreting
Visualizations
16
Key
Concepts

17. @paigerduty
Understand
Telemetry
Types

18. @paigerduty
Telemetry Types
18

Logs

Metrics

Traces

Events

19. @paigerduty 19

20. @paigerduty
EVENTS
-Deployments

-Alerts

-Feature Flags

-Infrastructure Updates

20

21. @paigerduty
METRICS
21
-Rolling up

-Recording rules

-Deriving

22. @paigerduty
TRACES
-Sampling strategy

-Retention period

-Adding metadata



22

23. @paigerduty
LOGS
23
-Indexing

-Structuring

-Timezone alignment

-Querying unindexed
logs

24. @paigerduty
THE “HOW”
24

25. @paigerduty 25
Scenario
The o11y bill is too high and Ops has
been asked to “fix it ASAP”.....

You’ve identified the top contributing
services - who does the cleanup?

26. @paigerduty
Follow
Telemetry
Journey

27. @paigerduty 27

28. @paigerduty
Generate
28
Collect
Transform
Export
Store

29. @paigerduty
Generate
29
Collect
Transform
Export
Store

30. @paigerduty
Generate
30
Collect
Transform
Export
Store

31. @paigerduty
Generate
31
Collect
Transform
Export
Store

32. @paigerduty
Generate
32
Collect
Transform
Export
Store

33. @paigerduty
THE “HOW”
33

34. @paigerduty 34
Scenario
Developer added some fields to their
logs but isn’t able to find and query
them in their tooling. What gives??

35. @paigerduty
Deciphering
Visualizations

36. @paigerduty 36

37. @paigerduty 37

38. @paigerduty
THE “HOW”
38
NYT “What’s Going On In This
Graph?”
Cognitive Apprenticeship SRECon

39. @paigerduty 39
Scenario
Developers keep blaming issues on “the
network” and your team is catching heat.

How do you move forward?

40. @paigerduty
NOW WHAT?

41. @paigerduty
-Slawek Ligus, Effective Monitoring and Alerting
41
Accept the reality —
the system as
perceived is not the
system as found

42. @paigerduty
O11y Knowledge Check
Where are all
the places I can
send data?
What is the
retention period for
logs? Metrics?
What metadata
(tags/labels) are
added
automatically?
How do we
sample
distributed
traces?
Add a main point
42
When and where are
aggregations applied to
metrics?

43. @paigerduty
Resources
43
Design for How People Learn
Julie Dirksen



Better Onboarding
Krystal Higgins
Software Telemetry
Jamie Riedesel

44. @paigerduty
Paige Cruz

@paigerduty
●O11y Concepts EVERY Dev
Should Know (blog)

●Open source o11y tutorials at
https://o11y-workshops.gitlab.io/

●Monitorama June 10-12

●Production Scavenger Hunt



44

45. @paigerduty
Credits
Special thanks to all the
people who made and
released these awesome
resources for free:
•Presentation template by
SlidesCarnival
•Photographs by Pexels
•
Happy designing!
45