Amazon DevOps Guru for Serverless Applications at JAWS Pankration 2024

In this talk we’ll use a standard serverless application that uses API Gateway, Lambda, DynamoDB, SQS, SNS, Kinesis, Step Functions, Aurora (Serverless) (and other AWS-managed services). We'll explore how Amazon DevOps Guru recognizes operational issues and anomalies like increased latency and error rates (timeouts, throttling and increased latency). We will also explore DevOps Guru "Proactive Insights" which recognize configurational anti-patterns like missing failure destination on Kinesis Data Streams or DLQ on SQS or over-provisioning of AWS services like DynamoDB tables. We'll also integrate DevOps Guru with PagerDuty to provide even better incident management. We'll also investigate current shortcomings of the DevOps Guru service.

Amazon DevOps Guru analyzes data like application metrics, logs, events, and traces to establish baseline operational behavior and then uses ML to detect anomalies. The service uses pre-trained ML models that are able to identify spikes in application requests, so it knows when to alert and when not to.

1. Vadym Kazulkin| @VKazulkin | ip.labsGmbH
Amazon DevOps Guru for the Serverless applications
Vadym Kazulkin, ip.labs, JAWS Pankration , August 24 2024
Amazon DevOps
Guru for the
Serverless
Applications
1

2. Vadym Kazulkin| @VKazulkin | ip.labsGmbH
Contact
Vadym Kazulkin
ip.labs GmbH Bonn, Germany
Co-Organizer of the Java User Group Bonn
[email protected]
@VKazulkin
https://dev.to/vkazulkin
https://github.com/Vadym79/
https://de.slideshare.net/VadymKazulkin/
https://www.linkedin.com/in/vadymkazulkin
https://www.iplabs.de/

3. Vadym Kazulkin| @VKazulkin | ip.labsGmbH
DevOps Lifecycle
5 Amazon DevOps Guru for the Serverless Applications
c

4. Vadym Kazulkin| @VKazulkin | ip.labsGmbH8
AIOPs
ArtificialIntelligenceforITOperations(AIOps)istheprocessofusingmachine
learningtechniquestosolveoperationalproblems.ThegoalofAIOpsistoreduce
humaninterventionintheIToperationsprocesses.
Byusingadvancedmachinelearningtechniques,youcanreduceoperational
incidentsandincreaseservicequality.AIOpscanhelpyouwith:
•Predictincidentsbeforetheyhappen
•Classifynewincidentsandinsights
https://aws.amazon.com/devops-guru
Amazon DevOps Guru for the Serverless Applications

5. Vadym Kazulkin| @VKazulkin | ip.labsGmbH9
What is AWS DevOps Guru
AmazonDevOpsGuruoffersafullymanagedAIOpsplatformpoweredbymachine
learning(ML)thatisdesignedtomakeiteasytoimproveanapplication’soperational
performanceandavailability
DevOpsGuruhelpsdetectbehaviorsthatdeviatefromnormaloperatingpatternsso
youcanidentifyoperationalissueslongbeforetheyimpactyourcustomers
•increasedlatency
•errorrates(timeouts,throttles,CPU,memoryand,diskutilization)
•resourceconstraints(exceedingAWSaccountlimits)
https://aws.amazon.com/devops-guru
Amazon DevOps Guru for the Serverless Applications

6. Vadym Kazulkin| @VKazulkin | ip.labsGmbH10
Benefits of DevOps Guru
https://aws.amazon.com/devops-guru
Amazon DevOps Guru for the Serverless Applications

7. Vadym Kazulkin| @VKazulkin | ip.labsGmbH11
https://aws.amazon.com/devops-guru
How DevOps Guru work
Amazon DevOps Guru for the Serverless Applications

8. Vadym Kazulkin| @VKazulkin | ip.labsGmbH12
DevOps Guru is powered by pre-trained ML models
•Builtdomain-specific,single-purposemodelstoidentifyknownfailuremodesinstead
ofnormalmetricbehavior.
•DevOpsGurureliesonalargeensembleofdetectors—statisticalmodelstunedto
detectcommonadversescenariosinavarietyofoperationalmetrics.
•DevOpsGurudetectorsdon’tneedtobetrainedorconfigured.Theywork
instantlyaslongasenoughhistoryisavailable.
•Individualdetectorsworkinpreconfiguredensemblestogenerateanomalieson
someofthemostimportantmetrics:errorrates,availability,latency,incoming
requestrates,CPU,memory,anddiskutilization,amongothers.
https://aws.amazon.com/blogs/machine-learning/amazon-devops-guru-is-powered-by-pre-trained-ml-models-that-encode-operational-excellence/
Amazon DevOps Guru for the Serverless Applications

9. Vadym Kazulkin| @VKazulkin | ip.labsGmbH14
DevOps Guru pre-trained ML detectors with periodic behaviors
•Many metrics, such as the number of
incoming requests in customer-facing
APIs, exhibit periodic behavior.
•The purpose of the causal convolution
detector is to analyze temporal data
with such patterns and to determine
expected periodic behavior.
•When the detector infers that a metric
is periodic, it adapts normal metric
behavior thresholds to the seasonal
pattern.
https://aws.amazon.com/blogs/machine-learning/amazon-devops-guru-is-powered-by-pre-trained-ml-models-that-encode-operational-excellence/
Amazon DevOps Guru for the Serverless Applications

10. Vadym Kazulkin| @VKazulkin | ip.labsGmbH
DevOps Guru Example Application
15
https://github.com/Vadym79/DevOpsGuruWorkshopDemo inspired by https://github.com/aws-samples/serverless-java-frameworks-samples
Amazon DevOps Guru for the Serverless Applications

11. Vadym Kazulkin| @VKazulkin | ip.labsGmbH
How future of software developers may look like
16 Amazon DevOps Guru for the Serverless Applications

12. Vadym Kazulkin| @VKazulkin | ip.labsGmbH18
Monitoring & Alerting of the Serverless Applications
Amazon DevOps Guru for the Serverless Applications

13. Vadym Kazulkin| @VKazulkin | ip.labsGmbH19
Monitoring & Alerting of the Serverless Applications
Amazon DevOps Guru for the Serverless Applications

14. Vadym Kazulkin| @VKazulkin | ip.labsGmbH20
DevOps Guru Set Up
Amazon DevOps Guru for the Serverless Applications

15. Vadym Kazulkin| @VKazulkin | ip.labsGmbH21
DevOps Guru Set Up with AWS Organizations
https://aws.amazon.com/blogs/mt/how-to-easily-configure-devops-guru-across-your-organization-with-systems-manager-quick-setup/
Amazon DevOps Guru for the Serverless Applications

16. Vadym Kazulkin| @VKazulkin | ip.labsGmbH
DevOps Guru Dashboard
22 Amazon DevOps Guru for the Serverless Applications

17. Vadym Kazulkin| @VKazulkin | ip.labsGmbH
DevOps Guru Dashboard
23 Amazon DevOps Guru for the Serverless Applications

18. Vadym Kazulkin| @VKazulkin | ip.labsGmbH
DevOps Guru Reactive Insights
24 Amazon DevOps Guru for the Serverless Applications

19. Vadym Kazulkin| @VKazulkin | ip.labsGmbH
DevOps Guru: Recognize Operational Issues in DynamoDB
26 Amazon DevOps Guru for the Serverless Applications
c

20. Vadym Kazulkin| @VKazulkin | ip.labsGmbH
DevOps Guru Examples: DynamoDB Throttling
27
hey -q 20 -z 15m -c 20 -H "X-API-Key: XXXa6XXXX "
https://XXX.execute-api.eu-central
1.amazonaws.com/prod/products/1
Amazon DevOps Guru for the Serverless Applications
c

21. Vadym Kazulkin| @VKazulkin | ip.labsGmbH29
DevOps Guru Examples: DynamoDB Throttling
Amazon DevOps Guru for the Serverless Applications

22. Vadym Kazulkin| @VKazulkin | ip.labsGmbH30
DevOps Guru Examples: DynamoDB Throttling
Amazon DevOps Guru for the Serverless Applications

23. Vadym Kazulkin| @VKazulkin | ip.labsGmbH31
c
DevOps Guru Examples: DynamoDB Throttling
Amazon DevOps Guru for the Serverless Applications

24. Vadym Kazulkin| @VKazulkin | ip.labsGmbH32
DevOps Guru Examples: DynamoDB Throttling
Amazon DevOps Guru for the Serverless Applications

25. Vadym Kazulkin| @VKazulkin | ip.labsGmbH33
DevOps Guru Examples: DynamoDB Throttling
Amazon DevOps Guru for the Serverless Applications

26. Vadym Kazulkin| @VKazulkin | ip.labsGmbH34
DevOps Guru Examples: DynamoDB Throttling
Amazon DevOps Guru for the Serverless Applications

27. Vadym Kazulkin| @VKazulkin | ip.labsGmbH35
DevOps Guru Examples: DynamoDB Throttling
Amazon DevOps Guru for the Serverless Applications

28. Vadym Kazulkin| @VKazulkin | ip.labsGmbH36
DevOps Guru Examples: DynamoDB Throttling
Amazon DevOps Guru for the Serverless Applications

29. Vadym Kazulkin| @VKazulkin | ip.labsGmbH37
DevOps Guru Examples: DynamoDB Throttling
Amazon DevOps Guru for the Serverless Applications

30. Vadym Kazulkin| @VKazulkin | ip.labsGmbH38
DevOps Guru Examples: DynamoDB Throttling
Amazon DevOps Guru for the Serverless Applications

31. Vadym Kazulkin| @VKazulkin | ip.labsGmbH41
•API Gateway 5xx and 4xx (throttling and not found) errors
•Lambda errors (throttling, timeouts, initerrors, increased latency)
•SQS with Lambda poller(message in flight stays to long in the
queue, due to error in Lambda)
•DynamoDB/Kinesis Streams and Lambda (high number of
asynchronous Lambda retries, due to error in Lambda)
•StepFunctionsinvoking Lambda (high number of Error State and
Lambda retries, due to error in Lambda)
•RDS and Aurora (Serverless v2) (high CPU load and highly increased
number of the database connections)
Other types of operational insights in the Serverless
applications recognized by DevOps Guru
Amazon DevOps Guru for the Serverless Applications

32. Vadym Kazulkin| @VKazulkin | ip.labsGmbH78
DevOps Guru Proactive Insights
Amazon DevOps Guru for the Serverless Applications

33. Vadym Kazulkin| @VKazulkin | ip.labsGmbH88
DevOps Guru integration in Incident Management Tools
•AWS OPsCenter(via AWS Systems Manager)
•PagerDuty
•Atlassian Opsgenie
Amazon DevOps Guru for the Serverless Applications

34. Vadym Kazulkin| @VKazulkin | ip.labsGmbH98
$3,024 per
resource per month
$2,016 per
resource per month
DevOps Guru Supported Services and Pricing
https://aws.amazon.com/de/devops-guru/pricing/
Amazon DevOps Guru for the Serverless Applications

35. Vadym Kazulkin| @VKazulkin | ip.labsGmbH99
DevOps Guru Cost Estimator
https://aws.amazon.com/de/devops-guru/pricing/
Amazon DevOps Guru for the Serverless Applications

36. Vadym Kazulkin| @VKazulkin | ip.labsGmbH100
DevOps Guru Conclusions, Obeservations, Suggestions
•Most operational issues have been correctly recognized so far
•It took several (at least 7) minutes to create an incident after
anomaly appeared
•Correctly no insights created for the temporary incidents
•Short time Lambda, DynamoDB and API Gateway Throttling
Amazon DevOps Guru for the Serverless Applications

37. Vadym Kazulkin| @VKazulkin | ip.labsGmbH101
DevOps Guru Conclusions, Obeservations, Suggestions
•Recommendations for the insight reason could be more precise (these
are limitations of CloudWatch though)
•No precise HTTP response code as API Gateway response but 4XX
and 5XX
•No differentiation between Lambda throttling because of reaching
individual function concurrency limit or the total AWS account
concurrency limit
•No differentiation between Lambda Timeout and Init Error
Amazon DevOps Guru for the Serverless Applications

38. Vadym Kazulkin| @VKazulkin | ip.labsGmbH105
•#AWS #Wishlist for DevOps Guru
•Support for EventBridge(and EventBridgePipes)
•Support for AppSync
•Support for Aurora Serverless v2 over Data API
•Better support for tracing i.e. AWS X-Ray, CloudWatch ServiceLens
and integrations with the 3
rd
observability tools i.e. Lumigo,
Datadog
DevOps Guru Conclusions, Obeservations, Suggestions
Amazon DevOps Guru for the Serverless Applications

39. Vadym Kazulkin| @VKazulkin | ip.labsGmbH107
Thank you
Amazon DevOps Guru for the Serverless Applications