KCD Porto: Choose Your Own Adventure - Cloud Naive Observability Pitfalls

Are you looking at your organization's efforts to enter or expand into the cloud native landscape and feeling a bit daunted by the vast expanse of information surrounding cloud native observability? When you're moving so fast with agile practices across your DevOps, SRE's, and platform engineering teams, it's no wonder this can seem a bit confusing. Unfortunately, the choices being made have a great impact on both your business, your budgets, and the ultimate success of your cloud native initiatives. That hasty decision up front leads to big headaches very quickly down the road. In this talk, I'll introduce the problem facing everyone with cloud native observability followed by 3 common mistakes that I'm seeing organizations make and how you can avoid them!

Key takeaways - This session is never the same twice as you the audience / attendees choose from a list of cloud native observability pitfalls that DevOps have to contend with in their daily cloud native lives! Super engaging and fun to tour the challenges that interest you most!

1. chronosphere.io Choose Your Own Adventure Eric D. Schabell Director Evangelism @ericschabell{@fosstodon.org} KCD Porto, 27-28 Sep 2024 Cloud Native Observability Pitfalls

2. Cloud Native Observability

3. Cloud Native

4. Data volume Experiment: Hello World app on 4 node Kubernetes cluster with Tracing, End User Metrics (EUM), Logs, Metrics (containers / nodes) 30 days == +450 GB

5. Retention Retention Retention Retention Retention Retention Retention

6. Cloud Native at Scale

7. Observability…

8. Cloud Native Observability at Scale

9. O11y at Scale (need)

10. Picking Your Pitfalls Ignoring existing landscape Focusing on The Pillars Sneaky sprawling mess Controlling costs T he protocol jungle Underestimating cardinality (Click on a pitfall to jump to that section)

11. Ignoring existing landscape

12. If they can’t see me… they can’t hurt me ...

13. Prometheus for metrics, alerting, queries

14. Prometheus auto discovery

15. Manual instrumentation (java client lib)

16. Short link: bit.ly/prom-workshop

17. Applications (Java) OTel Auto Instrumentation (libraries) OTel API OTel SDK OTel Collector OTLP OTLP OTLP OpenTelemetry (Auto) instrumentation

18. Host Observability Backend (Prometheus, Jaeger, Fluent Bit, etc.) , Applications OTel Auto Instrumentation OTel API OTel SDK OTel Collector Agent OTLP OTLP OTLP OTLP OTLP OpenTelemetry Collector (agent)

19. Host Host Host Observability Backend (Prometheus, Jaeger, Fluent Bit, etc.) , Applications OTel Auto Instrumentation OTel API OTel SDK OTel Collector Agent OTLP OTLP OTLP OTLP Collector (gateway) OTel Collector Gateway

20. Short link: bit.ly/opentelemetry-workshop

21. Ignoring existing landscape Focusing on The Pillars Sneaky sprawling mess Controlling costs T he protocol jungle Underestimating cardinality (Click on a pitfall to jump to that section, or jump to end ) Picking Your Next Pitfall

22. 2. Focusing on The Pillars

23. Pillars Phases

24. Developer Technology Bottom up

25. Pillar problems…

26. Car is on fire…

27. Better outcomes… Faster remediation… Easier detection… Happier customers…

28. Phase 1 Know something is happening as fast as possible…

29. Phase 2 Triage with specific information…

30. Phase 3 Understand to ensure never happens again…

31. Ignoring existing landscape Focusing on The Pillars Sneaky sprawling mess Controlling costs T he protocol jungle Underestimating cardinality (Click on a pitfall to jump to that section, or jump to end ) Picking Your Next Pitfall

32. 3. Sneaky sprawling mess

33. Over 66% of organizations use more than 10 different observability tools – ESG report over exploding data volumes

34. Know Triage Understand

35. Ignoring existing landscape Focusing on The Pillars Sneaky sprawling mess Controlling costs T he protocol jungle Underestimating cardinality (Click on a pitfall to jump to that section, or jump to end ) Picking Your Next Pitfall

36. 4. Controlling costs

37. “It’s remarkable how common this situation is, where an organization is paying more for their observability data, than they do for their production infrastructure.” ?

38. O11y data storage costs are broken. Keeping everything model?

39. Know the cost of observability metrics data?

40. Control costs and improve productivity Observability Platform DATA COLLECTION CONTROL PLANE STORE LENS Telemetry Pipeline Reduce Enrich Secure TRANSFORM AND ROUTE DATA IN YOUR ENVIRONMENT STORE DATA IN THIRD PARTY LOG & SIEM SOLUTIONS

41. Chronosphere named a Leader in the 2024 Gartner® Magic Quadrant™ for Observability Platforms Gartner, Magic Quadrant for Observability Platforms: By Gregg Siegfried, Padraig Byrne, Mrudula Bangera, Matt Crossley (12 August 2024) GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally, and MAGIC QUADRANT is a registered trademark of Gartner, Inc. and/or its affiliates and are used herein with permission. All rights reserved. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose https://chronosphere.io/2024-gartner-magic-quadrant

42. Ignoring existing landscape Focusing on The Pillars Sneaky sprawling mess Controlling costs T he protocol jungle Underestimating cardinality (Click on a pitfall to jump to that section, or jump to end ) Picking Your Next Pitfall

43. 5. The protocol jungle

44. Without open standards, you’ll not find a way back…

45. Host Observability Backend (Prometheus, Jaeger, Fluent Bit, etc.) , Applications OTel Auto Instrumentation OTel API OTel SDK OTel Collector Agent OTLP OTLP OTLP OTLP OTLP OpenTelemetry Collector (agent)

46. Prometheus for metrics, alerting, queries

47. Ignoring existing landscape Focusing on The Pillars Sneaky sprawling mess Controlling costs T he protocol jungle Underestimating cardinality (Click on a pitfall to jump to that section, or jump to end ) Picking Your Next Pitfall

48. 6. Underestimating cardinality

49. The struggle is real “I don't yet collect spans/traces because I can hardly get our devs to care about basic metrics, let alone traces.” “This is a large enterprise with approx. 1000 developers. Cultivating a culture of engineering that cares about availability is a challenge that we need to solve alongside any technical implementations.”

50. 10 hours on average, per week, trying to triage and understand incidents - a quarter of a 40 hour work week

51. 33% said those issues disrupted their personal life 39% admitting they are frequently stressed out

52. Cloud Native Observability at Scale

53. Control costs and improve productivity Observability Platform DATA COLLECTION CONTROL PLANE STORE LENS Telemetry Pipeline Reduce Enrich Secure TRANSFORM AND ROUTE DATA IN YOUR ENVIRONMENT STORE DATA IN THIRD PARTY LOG & SIEM SOLUTIONS

54. Ignoring existing landscape Focusing on The Pillars Sneaky sprawling mess Controlling costs T he protocol jungle Underestimating cardinality (Click on a pitfall to jump to that section, or jump to end ) Picking Your Next Pitfall

55. What should be the #1 item on cloud wishlist ? What should be #1 item on your cloud native observability wishlist?

56. chronosphere.io Questions? Eric D. Schabell Director Evangelism @ericschabell{@fosstodon.org}