The "Zen" of Python Exemplars - OTel Community Day

The Zen of Python states "There should be one-- and preferably only one --obvious way to do it." OpenTelemetry is the obvious choice for traces but bad news for Pythonistas when it comes to metrics because both Prometheus and OpenTelemetry offer compelling choices. Let's look at all of the ways you can tie metrics and traces together with exemplars whether you're working with OTel metrics, Prom metrics, Prom-turned-OTel metrics, or OTel-turned-Prom metrics!

1. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
The “Zen” {
Paige Cruz, Chronosphere
[of Python Exemplars]
}

2. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
01Exemplars Explained
02Example Exemplars
03Exemplar Enlightenment
Table Of ‘Contents’ {
}
talk.json
00The Zen of Python

3. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
00 {
[The Zen of Python]
}

4. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
hello.py

5. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
The Zen of Python (aka PEP 20)

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

6. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
…

There should be one-- and preferably only
one --obvious way to do it.

…

The Zen of Python (aka PEP 20)

7. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
??????

8. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
01 {
[Exemplars Explained]
}

9. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14

10. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14

11. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14

12. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
02 {
Example Exemplars:
OpenTelemetry
}

13. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
— OpenTelemetry Metrics Data Model Spec
“An exemplar is a recorded value that
associates OpenTelemetry context to a
metric event within a Metric”
definition.txt

14. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Value
Filtered Attributes
< Unix Epoch nanoseconds >
Observation Timestamp
OpenTelemetry ‘Exemplars’{
}
Trace

15. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
OpenTelemetry Metrics {
}
Metric Point

Sum ✅
Gauge ✅
Histogram /
Exponential
Histogram
✅
Exemplars?

16. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14

17. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14

18. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
02 {
Example Exemplars:
Prometheus
}

19. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
— OpenMetrics Specification
< “Exemplars are references to data
outside of the MetricSet. A common use
case are IDs of program traces.” >
definition.txt

20. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
< Integer or 64-bit float >
< traceID, spanID <=128 UTF-8 chars >
Value
LabelSet
< Unix Epoch seconds >
Timestamp
OpenMetrics ‘Exemplars’{
}

21. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Prometheus Metrics {
}
Metric Type

Counter ✅
Histogram ✅
Gauge ❌
Summary ❌
Exemplars?

22. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
traces
metrics

23. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
def do_something():
with tracer.start_as_current_span("do_something") as span:
data = call_api()
filtered = parse_data(data)
trace_id = '{:032x}'.format(span.get_span_context().trace_id)
TASK_COUNTER.inc(1, exemplar={"trace_id": trace_id, "trace_url":
f"http://localhost:16686/trace/{trace_id}"})
return filtered

script.py

24. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
FROM prom/prometheus:v2.48.0

ADD prometheus.yml /etc/prometheus

ENTRYPOINT [ "prometheus" ]

CMD [ "--config.file=/etc/prometheus/prometheus.yml",
"--enable-feature=exemplar-storage " ]
Buildfile-prom

25. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14

26. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14

27. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14

28. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14

29. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
02 {
[Example Exemplars]
Prometheus < > OTel
}

30. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
musings.md

31. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
03 {
Exemplar Enlightenment
}

32. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Exemplars {
}
01Traces are discoverable in typical workflows
02Jumpstart good/bad trace comparisons
03Bridge for (potentially) siloed telemetry
the_good.txt

33. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14

34. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
OpenMetrics Spec
OTel Compatibility with Prom & OpenMetrics

35. @paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
CREDITS: This presentation template was
created by Slidesgo, including icons by
Flaticon, and infographics & images by Freepik
Thanks <3 {
paigerduty
@hachyderm.io
@chronopshere.io

www.paigerduty.com
Get in touch
}