Choose Your Own Adventure - Cloud Native Observability Pitfalls

Are you looking at your organization's efforts to enter or expand into the cloud native landscape and feeling a bit daunted by the vast expanse of information surrounding cloud native observability? When you're moving so fast with agile practices across your DevOps, SRE's, and platform engineering teams, it's no wonder this can seem a bit confusing. Unfortunately, the choices being made have a great impact on both your business, your budgets, and the ultimate success of your cloud native initiatives. That hasty decision up front leads to big headaches very quickly down the road. In this talk, I'll introduce the problem facing everyone with cloud native observability followed by 3 common mistakes that I'm seeing organizations make and how you can avoid them!

Key takeaways - This session is never the same twice as you the audience / attendees choose from a list of cloud native observability pitfalls that DevOps have to contend with in their daily cloud native lives! Super engaging and fun to tour the challenges that interest you most!

1. chronosphere.io Choose Your Own Adventure Eric D. Schabell Director Evangelism @ericschabell{@fosstodon.org} Cloud Native Observability Pitfalls

2. Cloud Native Observability

3. Cloud Native

4. Data volume Experiment: Hello World app on 4 node Kubernetes cluster with Tracing, End User Metrics (EUM), Logs, Metrics (containers / nodes) 30 days == +450 GB

5. Retention Retention Retention Retention Retention Retention Retention

6. Cloud Native at Scale

7. Observability…

8. Cloud Native Observability at Scale

9. O11y at Scale (need)

10. Picking Your Pitfalls Ignoring existing landscape Focusing on The Pillars Sneaky sprawling mess Controlling costs T he protocol jungle Underestimating cardinality (Click on a pitfall to jump to that section)

11. Ignoring existing landscape

12. If they can’t see me… they can’t hurt me ...

13. Prometheus for metrics, alerting, queries

14. Prometheus auto discovery

15. Manual instrumentation (java client lib)

16. Short link: bit.ly/prom-workshop

17. Applications (Java) OTel Auto Instrumentation (libraries) OTel API OTel SDK OTel Collector OTLP OTLP OTLP OpenTelemetry (Auto) instrumentation

18. Host Observability Backend (Prometheus, Jaeger, Fluent Bit, etc.) , Applications OTel Auto Instrumentation OTel API OTel SDK OTel Collector Agent OTLP OTLP OTLP OTLP OTLP OpenTelemetry Collector (agent)

19. Host Host Host Observability Backend (Prometheus, Jaeger, Fluent Bit, etc.) , Applications OTel Auto Instrumentation OTel API OTel SDK OTel Collector Agent OTLP OTLP OTLP OTLP Collector (gateway) OTel Collector Gateway

20. Short link: bit.ly/opentelemetry-workshop

21. Ignoring existing landscape Focusing on The Pillars Sneaky sprawling mess Controlling costs T he protocol jungle Underestimating cardinality (Click on a pitfall to jump to that section, or jump to end ) Picking Your Next Pitfall

22. 2. Focusing on The Pillars

23. Pillars Phases

24. Developer Technology Bottom up

25. Pillar problems…

26. Car is on fire…

27. Better outcomes… Faster remediation… Easier detection… Happier customers…

28. Phase 1 Know something is happening as fast as possible…

29. Phase 2 Triage with specific information…

30. Phase 3 Understand to ensure never happens again…

31. Ignoring existing landscape Focusing on The Pillars Sneaky sprawling mess Controlling costs T he protocol jungle Underestimating cardinality (Click on a pitfall to jump to that section, or jump to end ) Picking Your Next Pitfall

32. 3. Sneaky sprawling mess

33. Over 66% of organizations use more than 10 different observability tools – ESG report over exploding data volumes

34. Know Triage Understand

35. Ignoring existing landscape Focusing on The Pillars Sneaky sprawling mess Controlling costs T he protocol jungle Underestimating cardinality (Click on a pitfall to jump to that section, or jump to end ) Picking Your Next Pitfall

36. 4. Controlling costs

37. “It’s remarkable how common this situation is, where an organization is paying more for their observability data, than they do for their production infrastructure.” ?

38. O11y data storage costs are broken. Keeping everything model?

39. Know the cost of observability metrics data?

40. DATA COLLECTION CONTROL PLANE PURPOSE-BUILT DATA STORES PER TELEMETRY TYPE CHRONOSPHERE LENS Align cost to value Single Tenanted Architecture w/ 99.99% Reliability Turns raw data into generated insights for each user Customer Environment Chronosphere SaaS Platform METRICS | LOGS | TRACES | EVENTS Ingest all your data from any source

41. Ignoring existing landscape Focusing on The Pillars Sneaky sprawling mess Controlling costs T he protocol jungle Underestimating cardinality (Click on a pitfall to jump to that section, or jump to end ) Picking Your Next Pitfall

42. 5. The protocol jungle

43. Without open standards, you’ll not find a way back…

44. Host Observability Backend (Prometheus, Jaeger, Fluent Bit, etc.) , Applications OTel Auto Instrumentation OTel API OTel SDK OTel Collector Agent OTLP OTLP OTLP OTLP OTLP OpenTelemetry Collector (agent)

45. Prometheus for metrics, alerting, queries

46. Ignoring existing landscape Focusing on The Pillars Sneaky sprawling mess Controlling costs T he protocol jungle Underestimating cardinality (Click on a pitfall to jump to that section, or jump to end ) Picking Your Next Pitfall

47. 6. Underestimating cardinality

48. The struggle is real “I don't yet collect spans/traces because I can hardly get our devs to care about basic metrics, let alone traces.” “This is a large enterprise with approx. 1000 developers. Cultivating a culture of engineering that cares about availability is a challenge that we need to solve alongside any technical implementations.”

49. 10 hours on average, per week, trying to triage and understand incidents - a quarter of a 40 hour work week

50. 33% said those issues disrupted their personal life 39% admitting they are frequently stressed out

51. Cloud Native Observability at Scale

52. DATA COLLECTION CONTROL PLANE PURPOSE-BUILT DATA STORES PER TELEMETRY TYPE CHRONOSPHERE LENS Align cost to value Single Tenanted Architecture w/ 99.99% Reliability Turns raw data into generated insights for each user Customer Environment Chronosphere SaaS Platform METRICS | LOGS | TRACES | EVENTS Ingest all your data from any source

53. Ignoring existing landscape Focusing on The Pillars Sneaky sprawling mess Controlling costs T he protocol jungle Underestimating cardinality (Click on a pitfall to jump to that section, or jump to end ) Picking Your Next Pitfall

54. What should be the #1 item on cloud wishlist ? What should be #1 item on your cloud native observability wishlist?

55. chronosphere.io Questions? Eric D. Schabell Director Evangelism @ericschabell{@fosstodon.org}