Beginner's Guide to Observability@Devoxx PL 2024

....

1. BEGINNER'S GUIDE TO
BEGINNER'S GUIDE TOBEGINNER'S GUIDE TOBEGINNER'S GUIDE TOBEGINNER'S GUIDE TOBEGINNER'S GUIDE TOBEGINNER'S GUIDE TOBEGINNER'S GUIDE TOBEGINNER'S GUIDE TOBEGINNER'S GUIDE TOBEGINNER'S GUIDE TOBEGINNER'S GUIDE TO
OBSERVABILITY
OBSERVABILITYOBSERVABILITYOBSERVABILITYOBSERVABILITYOBSERVABILITYOBSERVABILITYOBSERVABILITYOBSERVABILITYOBSERVABILITYOBSERVABILITYOBSERVABILITY
Michał Niczyporuk
@mihn

2. ABOUT ME

3. WHAT IS
WHAT ISWHAT ISWHAT ISWHAT ISWHAT ISWHAT ISWHAT ISWHAT ISWHAT ISWHAT ISWHAT IS
OBSERVABILITY?
OBSERVABILITY?OBSERVABILITY?OBSERVABILITY?OBSERVABILITY?OBSERVABILITY?OBSERVABILITY?OBSERVABILITY?OBSERVABILITY?OBSERVABILITY?OBSERVABILITY?OBSERVABILITY?

4. OBSERVABILITY
OBSERVABILITYOBSERVABILITYOBSERVABILITYOBSERVABILITYOBSERVABILITYOBSERVABILITYOBSERVABILITYOBSERVABILITYOBSERVABILITYOBSERVABILITYOBSERVABILITY
amercanized: Observability -> o11y

5. THE DEFINITION
“In control theory, observability is a
measure of how well internal states of a
system can be inferred from knowledge of
its external outputs”
Wikipedia

6. LET'S ASK THE ORACLE

7. WHAT FOR?
Way of determining state of the system
Observe trends
Spot anomalies
Debug errors
Gather data to support decision process
Measure user experience

8. LOGGING
LOGGINGLOGGINGLOGGINGLOGGINGLOGGINGLOGGINGLOGGINGLOGGINGLOGGINGLOGGINGLOGGING

9. LOGGING PRIME DIRECTIVE
Don't use System.out.println()

10. LOGGING LIBRARY ECOSYSTEM

11. DEFINE LOGGER
public class UserService {
private static final org.slf4j.Logger LOGGER =
org.slf4j.LoggerFactory.getLogger(UserService. class);
(...)
}

12. USE LOGGER
private int myMethod(int x) {
LOGGER.info("running my great code with x={}" , x);
int y = methodX(x);
LOGGER.info("my great code finished with y={}" , y);
return y;
}
2023-03-21 19:24:43,829 [main] INFO UserService - running my gre
2023-03-21 19:24:43,829 [main] INFO UserService - my great code

13. "TALKING TO A VOID" DEBUGGING
private void thisCallsMethodX() {
LOGGER.info("PLEASE WORK");
methodY();
LOGGER.info("SHOULD HAVE WORKED" );
}

14. DOS
Log meaningful checkpoints
Use correct logging levels
Add ids to logging context:
e.g. correlation id, user id
High-throughput = async appender

15. DO NOTS??????
Logs can be lost
Logs are not audit trail
Logs are not metrics
Logs are not data warehouse/lake
Alerts based on logs are flimsy - metrics-based alerts
are better

16. WHAT NOT TO LOG
personal information, secrets, session tokens etc.
(thanks )
OWASP Logging Cheat sheet
@piotrprz

17. CENTRALIZED LOGS
CENTRALIZED LOGSCENTRALIZED LOGSCENTRALIZED LOGSCENTRALIZED LOGSCENTRALIZED LOGSCENTRALIZED LOGSCENTRALIZED LOGSCENTRALIZED LOGSCENTRALIZED LOGSCENTRALIZED LOGSCENTRALIZED LOGS

18. CENTRAL STORAGE
Software that can ingest, process and search the logs

19. PUBLISHING LOGS

20. LOGS FORMAT
Unified format (e.g. JSON, logfmt, OpenTelemetry
Protocol OTPL, vendor specific)
regexes = two problems

21. GROWTH RULE OF THUMB

22. TOOLING
Cloud providers:
AWS: Cloudwatch Logs
Azure: Monitor Logs
GCP: Cloud Logging
Elastic Stack = ElasticSearch + Logstash/Filebeat +
Kibana
Graylog
Grafana Loki

23. METRICS
METRICSMETRICSMETRICSMETRICSMETRICSMETRICSMETRICSMETRICSMETRICSMETRICSMETRICS

24. METRICS
numbers
indexed over time
and multiple additional dimensions
=== timeseries

25. STORAGE
timeseries database

26. PUBLISHING METRICS

27. FORMAT
Storage specific
Vendor neutral: OpenTelemetry

28. METRICS TYPES
Prometheus

29. COUNTER
Used for: requests count, jobs count, errors

30. GAUGE
Used for memory usage, thread/connection pools
count, etc.
Tip: Store actual current and maximum value, not the
percentages

31. HISTOGRAM
used for request latencies and sizes
can calculate any percentiles via query
tip: preconfigured bucket sizes = performance better

32. SUMMARY
Histogram - but with percentiles (e.g. p50, p75, p95)
Calculated client side - lighter then histograms
Caveat: calculated per scrape target

33. METRIC ATTRIBUTES

34. METRIC RESOLUTION
Main dimension
How often metrics are probed
Long-term storage = Down sampling/roll up
1min up to 7 days -> 10 min upto 30 days, etc.
YMMV - think about business cycles (e.g. Black Friday)

35. METRIC TAGS/LABELS
Additional dimensions
Metadata: HTTP attributes, service name, hosts, cloud
regions, etc.
Metrics labels don't like high cardinality
route="/users/:id" ✅
route="/users/2137" ❌

36. WHAT TO MEASURE?
Kirk Pepperdine's The Box model
Your framework/libraries cover some basics already

37. KIRK PEPPERDINE'S THE BOX
"People": incoming requests, messages, jobs...
Application (Netty, Tomcat, Spring, etc.): thread pools,
queue sizes, requests...
Runtime metrics (JVM): GC, memory usage, threads...
Hardware: CPU, memory, I/O, disk usage

38. HOW TO MEASURE?

39. HOW TO MEASURE?
Latency
Traffic
Errors
Saturation
The Four Golden Signals:

40. MEASUREMENT METHODS
RED vs USE

41. RED
Rate
Error
Duration
Application focused

42. USE
Utilization
Saturation
Errors
Infrastructure focused

43. ONE METRICS TO RULE THEM ALL
APDEX
https://www.apdex.org/
More user experience focused - mesures satisfaction
Single number

44. APDEX - SPLITTING THE POPULUS
Minimal sample size 100 - adjust time window
Measure as close to the user as possible

45. APDEX - CALCULATIONS
Result is a decimal between 0 and 1

46. APDEX - INTERPRETING THE RESULT
0.94 ≤ X ≤ 1 : excellent
0.85 ≤ X ≤ 0.93 : good
0.70 ≤ X ≤ 0.84 : fair
0.50 ≤ X ≤ 0.69 : poor
0.00 ≤ X ≤ 0.49 : unacceptable

47. APDEX - CAVEATS
Very generic metric - hides details
Should be monitored closely after deployments
Should measure one functionality
Shouldn't be only metric of application success

48. PERCENTILES
Latency distribution is not normal/Gaussian
Use high percentiles (p99+) - not averages or means
(p50)

49. DIFFERENCE OF OPINION?

50. COORDINATED OMISSION
1. Server mesures in wrong place
2. Requests can be stuck waiting for processing
Measure both clients and servers - if possible
"How not to measure latency" by Gil Tene

51. DEATH BY METRICS
Storing unnecessary amount of metrics
Made possible by automations
Bad for infrastructure and cloud bills
Bad for your mental health

52. ASSUMPTIONS ABOUT METRICSWRONG
1. Metrics can't be lost
2. Metrics are precise

53. GROWTH RULE OF THUMB

55. EXAMPLE - "MISSING" INDEX

56. TOOLS
OSS:
+
Libraries:
Cloud providers:
AWS: Cloudwatch Metrics
Azure: Monitor Metrics
GCP: Cloud Metrics
PrometheusGrafana
Thanos
Victoria Metrics
Elastic Stack
Micrometer - API abstraction
OpenTelemetry

57. DISTRIBUTED TRACING
DISTRIBUTED TRACINGDISTRIBUTED TRACINGDISTRIBUTED TRACINGDISTRIBUTED TRACINGDISTRIBUTED TRACINGDISTRIBUTED TRACINGDISTRIBUTED TRACINGDISTRIBUTED TRACINGDISTRIBUTED TRACINGDISTRIBUTED TRACINGDISTRIBUTED TRACING

58. TRACING
A
B
C
traceID = X
span ID = A
HTTP Request
processed
traceID = X
span ID = B
parent span ID =A
HTTP Request
processed
traceID = X
span ID = C
parent span ID =B
HTTP Request
processed
traceID = X
span ID = D
parent span ID =C
db query executed
traceID = X
span ID = E
parent span ID =A
HTTP Request
processed
traceID = X
span ID = F
parent span ID =E
HTTP Request
processed
traceID = X
span ID = G
parent span ID =F
redis query
executed
Internet

59. ANATOMY OF A TRACE

60. Single trace = multiple spans
Each span contains
Trace ID
Span ID
timestamp and duration
Parent span ID, if applicable
All the metadata with any
cardinality

61. STORAGE

62. PUBLISHING TRACES

63. TRACING IDS FORMAT
Propagated in headers (e.g.HTTP) or in
metadata/attributes (e.g. Kafka)
Newer:
Older:
W3C Trace Context
Zipkin's B3

64. TRACE PROPAGATION
Trace ID/Span ID propagation
exporting spans to storage

65. TRACE PROPAGATION IN SERVICES
your stack might be already doing that for you

66. TRACE PROPAGATION IN INFRASTRUCTURE

67. STORING TRACES

68. HEAD SAMPLING VS TAIL SAMPLING

69. HEAD SAMPLING
Decision made without looking at whole trace
Random sample of traffic - law of large numbers makes
its viable

70. TAIL SAMPLING
Decision made after considering all or most spans from
a trace
Allows for custom behaviour

71. TAIL SAMPLING

72. THROUGHPUT-BASED TAIL SAMPLING
samples up to defined number of spans per second

73. RULES-BASED TAIL SAMPLING
Storing traces matching custom policies
custom policies: spans with errors, slow traces, ignore
healthchecks/metric endpoints/websockets, etc.

74. DYNAMIC TAIL SAMPLING
Aims to have a representation of span attribute values
among collected traces in a timeframe
example: store 1 in 100 traces, but with representation
of values in attributes http.status_code,
http.method, http.route and service.name

75. MIXING SAMPLING METHODS
Sampling methods can and should be mixed

76. WHERE SAMPLING HAPPENS?

77. TYPICAL PROBLEMS IN TRACING

78. MISSING INSTRUMENTATIONS
investigation needed
add missing instrumentation
wrap with custom span

79. MISSING SPANS
tools are smart enough to detect that
buffers for spans are too small
...or you hit ratelimiter somewhere

80. GROWTH RULE OF THUMB

82. EXAMPLE

83. LAST TIP
Put trace and span ID's in logs

84. TOOLS
Storage and visualization:
Abstraction API:
Cloud providers:
AWS: X-Ray
Azure: Azure Monitor Logs
GCP: Cloud Trace
Zipkin
Jaeger
OpenTelemetry
Micrometer Tracing

88. WHAT IS OPENTELEMETRY
vendor neutral observability framework

89. EVENTS
aka "true" observability
aka Observability 2.0

90. WHAT ARE EVENTS?
Events == spans

91. WHAT'S DIFFERENT?
Events are the fuel
Columnar storage and query engine are the
superpower
They give ability to slice and dice data and discover
unknown unknowns

92. UNKNOWN UNKNOWNS
Things we don't know we don't know
Example: Migrating to OpenTelemetry and from logs to
traces

93. INSPIRATION
Scuba: Diving into Data at Facebook

94. THE O.G

95. How to Avoid Paying for Honeycomb

96. HONEYCOMB UI

97. THE NEW KID
Very comprehensive observability package
Distributed tracing module needs ClickHouse

98. COROOT DISTRIBUTED TRACING UI

99. DEBUGGING WITH HONEYCOMB
Based on Sandbox on their website

100. HEATMAP

101. SELECTION

102. BUBBLEUP

103. BUBBLEUP - BASELINE VS SELECTION

104. BUBBLEUP - ENHANCE!

105. BUBBLEUP - POSSIBILITIES

106. BUBBLEUP - CHOSEN ONE

107. BUBBLEUP - PARAMETER VALUE

108. BUBBLEUP - FILTERED HEATMAP

109. BUBBLEUP - SELECTING TRACE

110. CULPRIT TRACE

111. CULPRIT TRACE - ZOOM

112. CULPRIT TRACE - ZOOM

113. CULPRIT TRACE - SIMILAR SPANS

114. CULPRIT TRACE - SIMILAR SPANS

115. CULPRIT TRACE - DEPLOYMENT

116. CULPRIT TRACE - DEPLOYMENT

117. THINGS TO REMEMBER
Observability = iterative & continuous process

118. WHAT AFTER?
SLI and SLO
Alerting and on-call

119. THANK YOU!
THANK YOU!THANK YOU!THANK YOU!THANK YOU!THANK YOU!THANK YOU!THANK YOU!THANK YOU!THANK YOU!THANK YOU!THANK YOU!
QUESTIONS?