The Art of Event Driven Observability with OpenTelemetry

To Make sure we provide a reliable system to our users we have implemented observability on our critical applications, services to be able understand precisely what is really happening in our environment. To help us in our Observability journey, OpenTelemetry has provided a standard on the way we are able to produce and collect measurements in our environments. Since the stable support of Traces in OpenTelemtry, many organizations are utilizing traces to understand what is happening and where we are spending time in our applications. We usually try to have an end 2 end traces starting from the mobile device or the browser of our users to the Database. Building tracing in a microservices architecture is usually a straightforward concept where each services has dependencies. But for architectures using events or messages to exchange information between services, …. it’s another story. How should a distributed trace look like for event driven architectures where you potentially have hundreds of asynchronous paths to a single message? Does it make sense to create an end-2-end trace for everything? Observability is not a science but art. We all get the brushes, colours, paper, and we all try to design a painting representing our system. In messaging architecture there are not one solution for all. We need to understand our system to paint the right telemetry. Sometimes in Event driven architecture, the usage of span links will resolve our challenge by having several traces for a transaction, but all those traces will be attached with the help of span links. During this presentation we will explain:
- The various components of OpenTelemetry
• Share examples not useful traces from event driven architecture.
• The purpose of span links
• The usage of span links in event driven architecture.

1. The Art of Event-Driven Observability with OpenTelemetry Henrik Rexed Cloud Native Advocate | Dynatrace

2. Henrik Rexed Cloud Native Advocate 15+ years of Performance engineering Owner of Producer of

3. Get a gentle reminder on OpenTelemetry The various components How to produce traces Learn the various way of instrumenting EDA architecture See the value of Span links If you stay with me you will ...

4. Once Upon a Time

5. 24 years ago … My Manager taught me how to use tools to get system health (perfmon, top, nmon…etc). Web Server Health 1999

6. 20 years ago… With solution having history and providing metadata from our environment helped me to design a better visualization: Web Server Health 2004

7. 13 years ago… With the usage of APM solutions, distributed traces, metrics , it was easier to represent the situation: Web Server Health 2010

8. 10 years ago… By looking at the logs produced by our application and servers, the situation was ….a bit ….clearer Web Server Health 2014

9. What I wanted to represent What I wanted to represent

10. Our artistic tools

11. Observability pillars Logs Events Metric Observability Traces Profiling Exception

12. Why do we need several signals? Analysis Context

13. We isolate our signals Most of the Organizations tends to separate the usage and storage of : Logs Traces Profiling Metrics

14. OpenTelemetry

15. OTel provides a set of  APIs , libraries , and tools to capture  distributed traces ,  metrics , and logs  from your applications. What is OpenTelemetry (OTel)? Azure Kubernetes GCP const span =   tracer .startSpan( 'operation1’ ); span .setAttribute( ‘customer_level’ , ‘gold’ ); span .end(); Service Python Service Node.js OTel Collector (optional) OTel API + SDK

16. The main Component of OpenTelemetry Instrument Collector

17. OpenTelemetry the Standard for Observability Metrics Logs Traces Continuous Profiling

18. How to produce traces?

19. How to add OTel to your applications? Service Java OTel Collector (optional) Manual Instrumentation Service Python const span =   tracer .startSpan( 'operation1’ ); span .setAttribute( ‘customer_level’ , ‘gold’ ); span .end(); Service Node.js // Pre-instrumented libraries pip install opentelemetry-instrumentation opentelemetry-instrument python myapp.py // OTel agent java –javaagent: path/to/otel javaagent.jar \ -jar myapp.jar Semi-automatic/ Auto instrumentation Observability backend

20. What is a trace? A trace is made of Spans. The trace Context glue all the various spans into a trace. Span 1 Span 2 Span 3 Span 4 Span 5 Span 6 Span Name Context Parent_id Start_time End_time Atribute Events { " name ": "Hello-Greetings", " context ": { " trace_id ": "0x5b8aa5a2d2c872e8321cf37308d69df2", " span_id ": "0x5fb397be34d26b51", }, "parent_id ": "0x051581bf3cb55c13", ‘ status_code’ “ STATUS_CODE_OK ” " start_time ": "2022-04-29T18:52:58.114304Z", " end_time ": "2022-04-29T18:52:58.114435Z", " attributes ": { " http.route ": "some_route1" }, " events ": [ { " name ": "hey there!", " timestamp ": "2022-04-29T18:52:58.114561Z", " attributes ": { " event_attributes ": 1 } }, { " name ": "bye now!", " timestamp ": "2022-04-29T22:52:58.114561Z", " attributes ": { " event_attributes ": 1 } } ], }

21. Tracing Instrumentation Otel OpenTelemetry API Propagator TraceProvider SpanProcessor Sampler Exporter Ressource Span

22. Resource Resource is the identity of a process production Telemetry data. The Resource is key to name telemetry components in the Observability backends. There are standard attributes to define a resource : Attribute Type Description Required? Service.name String Name of the service Yes Service.namespace String Namespace of the service.name No Service.instance.id String No Service.version String Version number of the service No

23. Tracing Sampler AlwaysON AlwaysOff ParentBased TraceIdratioBased parentbased_always_on parentbased_traceidratio parentbased_always_off Service A Service B Service C Service D Service E Observability backend 50% 25% 50% 10% 10% 1000 requests = 1 request with End2End trace with spans from service E

24. What is propagation? Service A Service B Context propagation Inject extract

25. In a traditional micro service architecture

26. Distributed tracing in normal architecture Distributed tracing allow us to attach: Keep track on all the tasks of a given transactions With this visualization we can understand where we are spending time in our transaction flow ‹#›

27. Distributed tracing in EDA With OpenTelemetry, I can represent my transactions in various way : One Big Traces Separate my transactions in sub transactions. ‹#›

28. Example 1 – End2End trace

29. Our environnent Otel-demo OpenTelemetry Operator Default EDA https://github.com/isItObservable/tracing_eda

30. Are generated distributed traces useful?

31. Example 2 – The usage of Span Links

32. Span Links Span link is feature allowing to create a relationship between separated spans. ~2min ~200ms ~2min ~1m30 Span link Span link

33. Our environnent Otel-demo OpenTelemetry Operator Default EDA https://github.com/isItObservable/tracing_eda

34. Conclusion

35. Pros Cons Sampling decision Relation between Producer & Consumer Visualize traces Keep track on the consumers Pros/Cons

36. Take Away Instrumentation Make sure your code is agnostic using no Vendor library and exporter Observability Make sure the metrics produced has enough dimensions Produce logs with contextual information Add Span Attributes to your spans Creativity Understand your system Design the right Observability

37. Looking for educational content on Observability , Checkout the Youtube Channel : Is It Observable Is it observable

38. Henrik Rexed [email protected] @twitter website/blog url Thank you! Let’s connect.