Smoke & Regression Execution Dashboard- Demo.pptx

Smoke and Regression testing

1. Smoke & Regression Execution Dashboards Grafana - Cloud  1 2017 Phreesia Confidential Information

2. Introduction (Why Grafana?) Roles and Responsibilities Objective of having a Dashboard How to access these Dashboards Regression Results Dashboard Details and Demo Smoke Results Dashboard Details and Demo  Agenda 2 2017 Phreesia Confidential Information

3. Grafana is centralized observability tool and Implementing Smoke Regression monitoring Dashboards on Grafana brings QA Automation Metrics into this centralized tool and provides consolidated and real time graphical visualizations of Application health. Introduction 3 2017 Phreesia Confidential Information - Serguei Mikhailov

4. ROLES and RESPONsIBILITIES 4 2017 Phreesia Confidential Information QA Director Serguei Mikhailov QA Manager Vijay Chekuri QA Solution Architecture Ritesh Kawadkar QA Automation Team - Contributor Nandini Lekkala QA Automation Team - Handover Alexey Perepelyak

5. Grafana Dashboards of Smoke Regression metrics are Scalable Scalable with Cloud Infra and Retention policies Customizable (Advanced Dashboarding) Grafana’s rich Dashboarding, Visualization, Querying and Data Filtering capabilities facilitate enormous customization Realtime Data is pushed to loki soon as jobs complete automatically, hence latest data is available Realtime Trackable Just like Jira all Smoke/Regression results for all builds of all jobs are tracked, with added advantage of cloud capabilities Fast & Responsive Dashboards use filtering to query specific sets of results data, hence they are highly responsive and interactive Granular Test Meta data is captured in detail including failure stack traces along with fail/exception types, Testcase level dashboards can also be created for Health Trends and monitoring Centralized users can see volumes of tests failing for each failure/exception type in centralized approach Sizable to Requirement Dashboards can be created for any level, Suite, Team, Component, Test Case etc. Advantages with Grafana on Cloud 5 2017 Phreesia Confidential Information

6. To provide centralized quickly accessible real-time view of the automation test execution metrics for all executions on all the environments. Facilitate the details of the all-test failures in a consolidated approach that can be filtered by designated parameters. To Allow ability to easily retrieve past execution data for past release for better tracking and trends reporting Create possibility to compare the test execution information for micro and macro level analysis on failure trends from dashboard itself.  Enable ease of grouping failures by type/exception to help with volume level insights through visualization & prioritization on where/what kind of fixes are necessary. Objective of having a Grafana Dashboard  6 2017 Phreesia Confidential Information

7. Open Grafana and search for the dashboard “Regression Execution Dashboard” or “Smoke Execution Dashboard” . Select all the parameters to fetch the appropriate regression execution test results. Regression Dashboard Link : https://grafana-nonprod.shared-services.phreesia.services/d/b45b85f9-6bf6-40cf-b348-bb045d02a046/regression-execution-dashboard Smoke Dashboard Link : https://grafana-nonprod.shared-services.phreesia.services/d/d70d5f68-36c6-497f-8f1d-002345ec4bef/smoke-execution-dashboard?orgId=1&from=now-24h&to=now&var-Environment=z2&var-version=23.16&var-Tag=UI.Smoke.Z2 Steps to View Implemented Smoke/ Regression Execution dashboards 7 2017 Phreesia Confidential Information

8. Regression Dashboard 8 2017 Phreesia Confidential Information

9. Every execution of the Regression testcase job will trigger a downstream job named “ PushRegressionDataToLoki”   Downstream job collects the test execution details in the JSON format and push it to Loki through an API call. It creates a JSON   with the test execution information.   How Regression Results data flows from Jenkins to Loki

10. REGRESSION DATA to Grafana – Data flow 10 2017 Phreesia Confidential Information Job calls Branches Services as per provided threads and Pacing Querying Influx DB to fetch the results  Influx DB and Grafana connected with Datasource connection Filtering with required parameters and time interval Displaying results on the dashboard Performance Jobs Average Response Time ,90 th Percentile, Api Name metrics are sent to Influx.

11. REGRESSION JOB Sample JSON Data 11 2017 Phreesia Confidential Information

12. REGRESSION JOB Sample JSON Data – Failure CASE 12 2017 Phreesia Confidential Information

13. Regression Dashboard Walkthrough/Demo 13

14. Smoke Dashboard 14 2017 Phreesia Confidential Information

15. Every execution of the Smoke job for each environment triggers a downstream job named “ PushSmokeDataToLoki” . Downstream job collects the test execution details in the JSON format and push it to Loki through an API call. It creates a JSON for each testcase with the test execution information. Loki will save the logs to the " qa_automation_logs " tenant for further querying. How Smoke Results data flows from Jenkins to Loki 15 2017 Phreesia Confidential Information

16. Smoke DATA to Grafana – Data flow 16 2017 Phreesia Confidential Information Dashboard Job pushes test execution data of respective environment to Loki through API Call  Querying Loki to fetch the results  Loki provide test information for selected parameters within the time interval  Filtering with required parameters and time interval Displaying results on the dashboard Smoke Execution Multi Jobs PushSmokeDataToLoki Once smoke execution is done, the downstream job will be triggered qa_automation_logs

17. Smoke JOB Sample JSON Data 17 2017 Phreesia Confidential Information

18. Smoke Dashboard Walkthrough/Demo 18 2017 Phreesia Confidential Information

19. Dashboard Panels Description 19 2017 Phreesia Confidential Information

20. Dashboard Panel Details 20 2017 Phreesia Confidential Information Panels details – Cont. Weekly Failure Testcases Metrics panel shows the graphical representation of the test cases failed for at least once within last 7days. Failure Reasons Weekly Metrics panel shows the graphical representation of the different failure reasons for the failed testcases within last 7days.

21. Panel Details – Cont.. 21 2017 Phreesia Confidential Information

22. Panel Details – Cont.. 22 2017 Phreesia Confidential Information

23. Panel Details – Cont.. 23 2017 Phreesia Confidential Information Test Execution Details panel is table of detailed information about the testcase execution and sorted with Failure testcases first for the selected time interval.

24. Questions and Answers 24

25. Thank you! 25