Observability in Modern Applications.pptx

Observability in Modern Applications

1. Observability in Modern Applications

2. What We’ll Discuss Today Introduction to Observability Understanding Logs, Metrics, Traces, and Alerts Key Challenges in Observability Proposed Solutions and Best Practices 2

3. What is observability?

4. Monitoring Vs Observability Monitoring is the process of collecting data and generating reports on different metrics that define system health. Observability is a more investigative approach. It looks closely at distributed system component interactions and data collected by monitoring to find the root cause of issues. It includes activities like trace path analysis, a process that follows the path of a request through the system to identify integration failures. Monitoring collects data on individual components, and observability looks at the distributed system as a whole. 4

5. Logs, Metrics, Traces and Alerts Fundamental components of observability

6. Logs Logs capture discrete events within components, providing detailed activity records over time. While rich in information, log data is often larger and can cause processing challenges, particularly with verbose logging. Filtered log data helps understand the context and details of potential problems identified through metric telemetry. 6

7. Metrics Metrics serve to provide insights into the health and operations of components or systems, offering point-in-time measurements of specific sources. Despite their small size, metrics enable efficient collection even in large-scale systems, facilitating effective monitoring. Pre-aggregation of metrics within components reduces computational overhead and storage requirements, particularly useful for processing numerous metric time series. The efficiency of metric processing and storage makes them ideal for automated alerting, serving as a reliable source of health data for all system components. 7

8. Traces Logging offers a snapshot of discrete events, while tracing provides a broader, continuous view of an application's flow. Tracing aims at understanding program flow and data progression, often tracking a user's journey through an entire application stack. Tracing focuses on optimization rather than reactive troubleshooting, helping developers identify and address performance bottlenecks. When troubleshooting, tracing reveals detailed information such as function details, duration, parameters, and the depth of user interaction within functions. 8

9. Alerts Alerting systems enable teams to detect issues or anomalies proactively by setting alerts based on predefined thresholds or conditions. This real-time notification allows for prompt investigation and resolution Alerts can be based on static thresholds or dynamic patterns in metric data, including machine learning for anomaly detection. Analyzing alerting data and incident response metrics allows teams to refine configurations, optimize thresholds, and enhance system resilience, driving continuous improvement in monitoring effectiveness and operational efficiency. 9

10. Key Challenges in Observability A look at infra as a whole

11. Key Challenges Lack of meaningful Metrics Lack of efficient Monitoring Dashboards No traces. Good to use. helps to optimize performance Alerts: No proper Runbooks for each alerts Too many micro services 11

12. Key Challenges Lack of proper disaster recovery plans Lack of proper release management Lack of Performance testing Normalize the stack Remove redundant software and tools Periodic review of existing software and tools for improvement 12

13. Dedicated Platform team? Foundation Builders

14. Dedicated Platform team Specialized Expertise : A platform team consists of individuals with specialized skills and knowledge in building, maintaining, and optimizing the underlying infrastructure, frameworks, and tools used by other teams within the organization. Their expertise ensures that the platform is robust, scalable, and efficient, facilitating the development and deployment of applications. Tooling and Automation : A platform team develops and maintains tools and automation workflows to streamline development, deployment, and operations processes. They create reusable components, templates, and scripts that enable teams to build and deploy applications more efficiently and consistently. 14

15. Dedicated Platform team Specialized Expertise : A platform team consists of individuals with specialized skills and knowledge in building, maintaining, and optimizing the underlying infrastructure, frameworks, and tools used by other teams within the organization. Their expertise ensures that the platform is robust, scalable, and efficient, facilitating the development and deployment of applications. Tooling and Automation : A platform team develops and maintains tools and automation workflows to streamline development, deployment, and operations processes. They create reusable components, templates, and scripts that enable teams to build and deploy applications more efficiently and consistently. 15

16. Dedicated Platform team Security and Compliance : Ensuring the security and compliance of the platform is a critical responsibility of the platform team. They implement security best practices, monitor for vulnerabilities, and enforce compliance standards to protect data and mitigate risks. Performance Optimization: The platform team continuously monitors and optimizes the performance of the infrastructure and applications running on the platform. They identify bottlenecks, fine-tune configurations, and implement performance improvements to enhance the reliability and efficiency of the platfor m. 16

17. Dedicated Platform team Support and Troubleshooting: Inevitably, issues and challenges arise with the platform and its underlying infrastructure. The platform team provides support and troubleshooting assistance to other teams, helping them diagnose and resolve problems quickly to minimize downtime and disruptions . Innovation and Evolution: A dedicated platform team is responsible for driving innovation and evolution of the platform. They research and evaluate new technologies, identify opportunities for improvement, and lead initiatives to modernize and enhance the platform to meet changing business needs and technological advancements. 17

18. Thank you Aneesh Kumar @ aneeshep | [email protected]