OSCamp Kubernetes 2024 | A Tester's Guide to CI_CD as an Automated Quality Control System by Alex Pshe

We all know about the importance of automatic deployment of applications to production, but do we all know how to build an automated quality control system that continuously provides feedback on the application’s state and decides whether the application is ready for production rollout in terms of quality metrics? I have prepared a step-by-step algorithm for setting up such a system on your project, with specifics for web applications. I will talk about the best quality control practices at different levels, from linters to performance testing, from underwater problems to the ideal monitoring system.

1. A Tester's Guide to CI/CD
as an Automated Quality Control System

2. ALEXPSHE ALEXPSHE ALEX
● SDET at JetBrains

● 20+ fully automated
quality control systems

● Big fan of process
automation ??????

3. Get an understanding
of the high-level algorithm
for building a quality system
Main takeaway
*There will be no technologies and specific solutions for individual cases here.

4. THE PROBLEM IS ..
Typical view on CI/CD and testing
Automation
tests
Not part of
CI/CDOR
Automation
tests
Part of
CI/CD
Green/Red
NOT BAD, BUT…

5. CI/CD -> Automated Quality Control System
New feature developed
Quality Metrics measured
Responsible people notified


Next process started


Quality Gates called


Pipeline artifacts collected


Time to respond increased 34%
Compressed stored image
Allowed +10% Automessage to
#performance channel
Save node settings
and benchmark log
Request developer`s update

6. DELIVERY PROCESS
For feature:
●Requirements testing
●Functional testing
●Non-functional testing

For product version:
●UAT testing
●Alpha/Beta testing
●Functional testing
●Non-functional testing

For release:
●Configuration testing
●Compatibility checks
●Availability checks
WHAT PROCESSES DO YOU NEED?

7. PROCESSES`S MAP
New feature
New feature
New feature
UAT
Release

8. FOLLOW STEPS: FOLLOW STEPS:
●Write down ALL processes in your team
●Choose the most crucial, expensive and long
●Implement automatic testing control system

9. A TESTER`S GUIDE TO CI/CD
Preparation steps:
●Business Metrics
●Quality Metrics
●Quality Gates
Pipeline steps:
●Pipeline triggers
●Pipeline stages
●Pipeline criteria

Feedback steps:
●Metric`s collecting
●System notifications
●Call to next process

FOR EACH DELIVERY PROCESS ->
Follow steps

10. LET'S IMAGINE
WE ARE TIKTOK`s TESTERS

11. FEATURE ASSESSMENT
New feature
LET`S SETUP PIPELINE FOR

12. PREPARATION STEPS

13. PREPARATION STEPS QUALITY SYSTEM
WHERE TO START?

14. PREPARATION STEPS 1.BUSINESS METRICS
WHY
Do we need
test automation?
Because:
We have to
decrease
manual testing
We have to
automate
test cases
We are
test automation
engineers

15. PREPARATION STEPS 1.BUSINESS METRICS
WHY
Do we need
test automation?
Business goal

Business metrics

Product metrics

More money More paid users+25% of retention
QA Auto as a tool to achieve product metrics
Then achieveThen achieve

16. PREPARATION STEPS 2.QUALITY METRICS
Product metrics

Quality metrics

should be
based on

17. PREPARATION STEPS LIST OF METRICS
LIST OF METRICS
Product metrics

Business Metric: Revenue Growth

Goal:
20% Overall Revenue Growth
for the Company
Over the Quarter



Product Metric: User Retention

Goal: User retention after first use
increases from 25% to 40% following
the implementation of:

feature that recommends content based
on the longest videos watched from the
last three


Business metrics

18. PREPARATION STEPS LIST OF METRICS
LIST OF METRICS
Product metrics

Quality metrics

Quality Metric: Test Coverage
for New Recommendation Feature

Goal: Achieve 90%
test coverage for the new video
recommendation feature.

Product Metric: User Retention

Goal: User retention after first use
increases from 25% to 40% following
the implementation of:

feature that recommends content based
on the longest videos watched from the
last three

19. PREPARATION STEPS LIST OF METRICS
LIST OF METRICS
Quality metrics

FROM NON FUNCTIONAL REQUIREMENTS
Quality Metric: Performance

Goal: Ensure TikTok can
maintain smooth performance
during peak times, handling
up to 1 million concurrent
users without degradation in
service quality.

20. PREPARATION STEPS 3. QUALITY GATES
based on Quality metrics

Quality gates

21. PREPARATION STEPS LIST OF METRICS
LIST OF METRICS
Quality metrics

Quality gates

https://pshe-academy.com/quality-metrics/en
Quality Metric: Test Coverage
for New Recommendation Feature

Goal: Achieve 90%
test coverage for the new video
recommendation feature.

Quality Gates:

>= 90% - allowed
< 90% - not allowed

22. PREPARATION STEPS CONCLUSION
STEPS TO DO:

1.Ask for business goals for the next quater/year
2.Clarify product metrics with product owner
3.Build qulity metrics based on product metrics
4.Setup quality gates based on quality metrics

23. PIPELINE STEPS

24. PIPELINE STEPS 1. PIPELINE TRIGGERS
Git events:
●New Merge Request
●New Commit
●New changes in folder

Manual actions:
●Manual start
●Person`s approve
●Message in Slack

Previous step ended:
●Triggered
by another pipeline
WHAT THE FLAG OF PROCESS START?

25. FEATURE ASSESSMENT
New feature
Triggered by commit:
1 commit = 1 MR
1.Limited resources
2.Have to get feedback before merge

26. PIPELINE STEPS 2. PIPELINE STAGES
Functional testing:
●Unit tests
●Integration tests
●System tests
WHAT TYPES OF STAGES?
Code quality:
●Checkstyle
●Linters

Non-Functional testing:
●Performance tests
●Security tests
●Availability tests
Custom heuristics:
●Issue state
●Bug`s probability
Critical checks:
●Build
●Compilation
●Configuration

27. FEATURE ASSESSMENT
New feature
Triggered by commit:
1 commit = 1 MR
Build Unit testsE2E tests
Performance
Tests

28. PIPELINE STEPS 3. PIPELINE CRITERIA
“FAIL FIRST” approach:
Pipeline has to fail
when it's obvious that
the process is not met
requirements

Time of execution:
Pipeline has to be finished
in appropriate time
(time that does not affect
other processes)

Reliability:
Pipeline has to provide
only true-positive feedback
(low flakiness rate)
WHAT CRITERIA OF THE EFFECTIVE PIPELINE?

29. FEATURE ASSESSMENT
New feature
Triggered by commit:
1 commit = 1 MR
Build Unit testsE2E tests
Performance
Tests
Optimize
build chain
Set Up
parallel runs
Increase
stability
Prepare
Prod-like env

30. PIPELINE STEPS CONCLUSION
STEPS TO DO:

1.Find correct trigger/s for the process
2.Design pipeline stages based on metrics/checks
3.Prioritize stages following “fail first” approach
4.Measure time/flakiness rate
to confirm pipeline quality criteria

31. FEEDBACK STEPS

32. FEEDBACK STEPS
1. METRIC`S COLLECTING
Statistics from reports:
●Test Coverage
●Test Results
●History and metadata

Execution collectors:
●Retries
●Performance data
●Artifacts

Custom scripts:
●Issue status
●Deploy status
HOW TO COLLECT ALL METRICS?

33. FEATURE ASSESSMENT
New feature
Triggered by commit:
1 commit = 1 MR
Build Unit testsE2E tests
Performance
Tests
Build logs

Test Report

Coverage Map
App logs

App performance metrics
Test Measurements

34. FEEDBACK STEPS
1. METRIC`S COLLECTING
Ready-to-use solutions:
●Grafana
●Allure Reports
●Sonarqube

Custom Solutions:
●Notion Dashboard
●HTML report
●Data report tools

WHERE TO STORE ALL METRICS?

35. ALLURE REPORT
Test Coverage
for New
Recommendation Feature

36. GRAFANA
Availability checks

37. FEEDBACK STEPS
2. SYSTEM NOTIFICATIONS
Step #1:
Level of
escalation
Step #2:
Notification
channels
HOW TO NOTIFY RESPONSIBLE PEOPLE?

38. FEATURE ASSESSMENT
New feature
Triggered by commit:
1 commit = 1 MR
Build Unit testsE2E tests
Performance
Tests
#build-chain
All crushes
#tiktok-dev
Not passed

#tiktok-qa
Coverage < 90%
#performance-team
Cannot handle 1m users
SLACK BOT

39. FEEDBACK STEPS
3. CALL TO THE NEXT PROCESS
CALL THE NEXT PROCESS
Automatically

Manually

40. PROCESSES`S MAP
New feature
New feature
New feature
UAT
Release
WHAT IS THE NEXT PROCESS?

41. FEATURE ASSESSMENT
New feature
LET`S SETUP PIPELINE FOR
AFTER
QA APPROVE

42. FEEDBACK STEPS CONCLUSION
STEPS TO DO:

1.Choose the best way to collect all metrics
2.Figure out responsibilities and call of actions
3.Setup a call to the next process based on
processes` map

43. AS A RESULT

44. AUTOMATED QUALITY CONTROL SYSTEM
1.Prepared stage priority
2.Automated metrics calculation
3.Implemented instruction
4.Collecting data and call next step
CI/CD as tester`s tool

Continuous feedback about
system quality status

45. SLIDES & QUESTIONS THANKS! THANKS! THANKS!