CONTINUOUS PERFORMANCE TESTING - ISTSTC (1).pdf

In today's fast-paced digital world, the performance of a system can make or break the user experience. Whether it's a website, mobile app, or software application, users expect lightning-fast response times and seamless functionality. As testers, it's our job to ensure that these expectations are met, and one of the most effective ways to do that is through Continuous Performance Testing (CPT).

1. Mesut Güneş
Principal Consultant @Modus Create, LLC.
Continuous Performance Testing - (CPT)
Motto: Performance is a living thing

2. 2
CONTINUOUS
PERFORMANCE TESTING
Performance is a living thing

3. 3
MESUT GÜNEŞ
PRINCIPAL CONSULTANT AT MODUS CREATE
in/gunesmes
gunesmes
story/gunesmes
gunesmes
testrisk.com
[email protected]

4. WHO YOU ARE
4?
Testers/QAs or SDETs
Developer
Performance engineer
Did performance testing

5. 5
TALK INCLUDE THESE TOPICS
Performance testing
Type of performance testing (load, stress, soak, volume, capacity, spike)
Automation, Tools, Technologies,
Benefits of Automated Tests
Integration
Thresholds, Percentile, rps
CI/CD
API
Orchestration
. . . and more

6. 6
TABLE OF
CONTENTS
PERFORMANCE
TESTING
Concept of performance
CONTINUOUS
PERFORMANCE
TESTING
Automation in performance
01
03
02
04
05
METRICS
Without data nothing
measurable
TOOLS AND
TECHNOLOGIES
What kind of tools we have
for this purpose
BENEFITS
What kind of benefits we
can have

7. 01
7
WHAT IS
PERFORMANCE
TESTING

8. PERFORMANCE TESTING
BY DEFINITION
8
Performance Testing is
+A non-functional testing
+Testing the functionalities of the system
+Both black-box and white-box testing
+Defining how the system works under any loads
+Applied under
○different load levels
○different load increments
○different durations
Defining the performance of the system by testing the
functionalities of the system in terms of the non-functionalities

9. 9
PERFORMANCE TESTING BY PROCESS
Performance testing should be also executed in a process. Performance engineers
should ensure that they collected requirements, created correct amount of virtual users,
created test scenarios and they can execute them. As a final they should check the
result and the system, if required after fixing, re-execution may be needed.
Planning
Creating test Scripts
Creating the Test Env.
Executing the Tests
Monitoring the Systems
Analyzing & Optimizing

10. 3 criterias
+Load - #VU
+Increment of the load
+Duration - Mins, Hours
TYPES OF
PERFORMANCE TESTING
10
Types of performance testing depend on the strategy
that applied during the test by means of the loads,
the duration and increment of the load

11. LOAD TEST
11
Load Test is
+Basic form
+Widely known-form
+Under optimum load
+Giving a result that users might be facing
Load test is to understand the behavior of the system under a
specific load. It is performing under specific amount of the load
to check if the system gives the expected responses with
keeping the load stable for a long term

12. STRESS TEST
12
Stress Test is
+Testing the upper boundary of the system
+Needing more resources
+Needing more investigation for boundaries
+Needing more attention
+Prone to break the system, aka: soak, endurance
testing
+good for promotion like black friday, 11.11
Stress test is to test the upper boundary of the system.
When the load is increased to the level of upper boundary,
test aims to get the response of the system under stress.

13. SOAK-ENDURANCE TEST
13
Soak Test is
+Testing the upper boundary of the system
+Testing the system for a prolonged time
+Needing more resources
+Needing more investigation for boundaries
+Needing more attention
+Aims to break the system
+Preparation for campaigns like black friday, 11.11
Soak test is to test the upper boundary of the system for a
prolonged time. Soak is good if you are not sure the duration of
the user for some occasions.
Eliud Kipchoge broke the 2-hour-barrier and changed the history of modern marathons in Vienna in 2019.
Image: https://images.app.goo.gl/6FCD6U95h8jFWuXX9

14. SPIKE TEST
14
Spike Test is
+Testing the sudden increase and decrease of loads
+Testing the scalability
+Requiring more load generations
+depends of the tools (load the system, then
spike it)
+Good for some occasion like simulating
+push notifications for e-coms
+announcing critical news
Spike test is to test the behaviour of the system when the
amount of load is suddenly increased and decreased. It aims
to find system failures when loads is changing to the
unexpected amount.

15. 15
CAPACITY TEST
Capacity Test is
+Testing the max user
+Finding the behavior when more users join
+Providing info about the loss of user data
+Ex: gradually increase the number of concurrent
users accessing a website and observe the server's
performance
Capacity testing is aiming to find the maximum amount of the
user that the system can handle. By this test we can find the
how many user can use the system.

16. 16
VOLUME TEST
Volume Test is
+Testing under huge amount of data receives
+Any issue with caching
+Database has any issue for indexing
+Ex: "Black Friday" sale scenario with a massive
surge of users accessing the website
simultaneously, check the scalability
Volume test is testing the performance when the system
receives huge amount of data. With the volume testing, we can
find issues for scalability.

17. 17
ALL-IN-ONE IMAGE
Reference: https://queue-it.com/blog/types-of-performance-testing

18. 18
WHAT IS
CONTINUOUS
PERFORMANCE
TESTING
02

19. 19
CONTINUOUS PERFORMANCE TESTING
CPT is designing the performance test to run
them spontaneously by tools and evaluation the
result by tools to decide to go or not to go
further as part of the CI/CD pipeline.

CPT is
ᐩPart of CI/CD
ᐩShift-left and/or shift-right approaches
ᐩSubset of the performance test scenarios
ᐩDesigned to run spontaneously by tools
ᐩDesigned to evaluate the result by tools
ᐩDesigned not to break the system
ᐩHaving decision metrics
ᐩDesigned to create report
ᐩPERCEPTIONAL
Reference: https://www.gocd.org/2016/06/08/Add-performance-testing-to-your-Continuous-Delivery-pipeline

20. 20
WHAT TYPES TO AUTOMATE
Continuous performance testing can be applied to the
all types of performance testing. However each types
need different level of maturity and sanity.
Automated Performance Testing is
Easily applied to load testing
Very hard to apply Soak-Endurance test
Hard to apply stress test, spike test, but have benefits

21. 21
AUTOMATED LOAD TESTING
Since the load test is aiming to test
the system under a specific load and
it is success is belong to the
predefined metrics, load testing is
the best suited performance testing
type for CPT.
Automated Load Testing is
+Needing specific load level
+Needing pre-defined metrics
+Needing evaluation of the result
+Needing exactly the same environment for every run
+Needing tools and script knowledge
+Easy to integrate CI/CD pipeline - everything is scripted
+Reporting the performance of the system when
something changes

22. 22
METRICS
03

23. 23
METRICS
Some of the Metrics are
+Response Metrics
+Response time
+Average response time
+Peak response time
+Wait time
+Error rate
Performance testing is meaningful when the metrics are
understandable for each counterparts. These metrics are
the key identifier for evaluating the results and deciding if
the test is successful or failed.
+Volume Metrics
+Concurrent users
+Requests per second
+Throughput

24. 24
RESPONSE METRICS
Variants of the Response Metrics are
+Response time
+Time to complete a request to the
system
+Time to First Byte (TtFB) -
response starts
+Time to Last Byte (TtLB) -
response ends
+Wait time is the time between
sending request to TtFB
Response time is the main measurement taken during the
test. Then we calculate the average, mean, median,
minimum, maximum values of it.
+Average, Median, Peak
response time
+Average is mean of the
whole responses
+Median is the one in
middle between min to
max
+Peak is the maximum
value
+Error Rate
+# of Error / # of Request

25. VOLUME METRICS
Volume related metrics are depends on the loads which are
generated during the tests by the tools. These metrics show how
much loads was sent to the system and how much response was
receipt. Basically these are requests-per-seconds (RPS),
concurrent user, throughput, hatch rate
Variants of the Volume Metrics are
+RPS is the number of request sending per second
+Concurrent User is the number of virtual users
which are testing the system

+Hatch Rate is the number of virtual user adding to
the system per second
+Throughput is the data in kilobytes which are sent
from the servers

26. 26
EVALUATION - NORMAL DISTRIBUTION
+Enough data → Normal distribution
+2 SD (standard deviation), 95
confidence level
+Remove outliers
+Test the hypothesis
H1: every metrics is similar with the
historical data within 95 confidence
level

27. 27
EVALUATION - THRESHOLDS - STATIC DATA
Thresholds is designed to define fail/pass criteria. With it you can
define max, min, average, median, throughput and other metrics.

28. 28
EVALUATION - HISTORICAL DATA - AI HELPS
We have to evaluate the response
metrics statistically.
Keeping the historical data as a base
and tracking the trends are critical.

29. 29
EVALUATION - HISTORICAL DATA - AI HELPS
Metrics in the historical data shows us
the steady-state of the system.
We can compare the current metrics
statistically with the history.
Hypothesis 1: Current response is within
the 95 confidence level
Hypothesis 2: Current response is not
within the 95 confidence level

30. 30
TOOLS AND
TECHNOLOGIES04

31. 31
TOOLS FOR CPT
Volume related metrics are depends on the loads which are
generated during the tests by the tools. These metrics show how
much loads was sent to the system and how much response was
receipt. Basically these are requests-per-seconds (RPS),
concurrent user, throughput, hatch rate
Tools Needed for Automated Performance Testing are
+Performance Testing tool
+Databases
+Reporting tools
+Comparison tools
+Orchestration Tools
+Monitoring

32. 32
PERFORMANCE TESTING TOOLS
We have a variety of tools in the industry, they changes from open
source to commercial and/or commercial support; newly started
to 20+ years old performance testing tools.
Some of the well-know Performance Tools are
+Jmeter - Apache project, v1.0 1998
+Gatling - Enterprise support by Frontline
+K6 - Enterprise support by LoadImpact
+Locust
+and many others
+
+
+LoadRunner by Microfocus (previously Mercury - HP)
+Blazemeter - support multiple opensource tools
+LoadUI Pro by Smartbear
Opensource Commercial

33. PROPRIETARY & CONFIDENTIAL 33
AUTOMATION TOOLS
We need to manage these tools to run the test efficiently. Every
CI/CD run on a orchestration tool that provide every needed task
by using different tools and scripts.
Tools Needed for Automated Performance Testing are
+Jenkins
+TravisCI
+TeamCity
+CircleCI
+Code Ship
+Gitlab CI

34. PROPRIETARY & CONFIDENTIAL 34
ORCHESTRATION TOOLS
We need to create test environments, some environment for
running performance testing and also some environment to
keeping data and reports. These environments should be created.
Some of the well-know Orchestration Tools are
+Docker Compose
+Kubernetes
+Prometheus
+Mesosphere DC/OS
+And others

35. PROPRIETARY & CONFIDENTIAL 35
MONITORING TOOLS
We need to track, measure, and analyze the performance of their
software applications, servers, networks, and other IT
infrastructure components
Some of the well-know Orchestration Tools are
+New Relic
+Dynatrace
+AppDynamics
+Datadog
+Nagios
+Zabbix
+Prometheus
+Grafana

36. 36
BENEFITS
05

37. When the benefits of continuous testing composed of the benefits of performance testing it becomes
more delicious :)

Some benefits of automated performance testing are
+Quick feedback - shift-left approach
+Giving current performance of the app
+Creating awareness about the performance of the system
+Initiate performance based development
+Preventing last minute mistake
+Giving history about the performance of the system
+Making it easy to run performance testing on demand




Image: https://images.app.goo.gl/TVy8DzqnVN4NhCHA8
BENEFITS

38. PROPRIETARY & CONFIDENTIAL 38
THANKS!
Questions?


References:
+https://www.software.broadcom.com/hubfs/ESD_FY22/DevOps/
Continuous-Performance-Testing-White-Paper_BC-Agile-2022_
CE-2377_v5B.pdf
+https://www.gocd.org/2016/06/08/Add-performance-testing-to-y
our-Continuous-Delivery-pipeline/
+https://www.anztb.org/userfiles/files/PerformanceTestingQLD041
4.pdf
+https://www.theseus.fi/bitstream/handle/10024/110354/Joonas_M
oilanen_Mervi_Jeskanen.pdf?sequence=1
+The Art of Software Testing,
http://barbie.uta.edu/~mehra/Book1_The%20Art%20of%20Softw
are%20Testing.pdf
+K6 manual, https://docs.k6.io/docs/welcome
+https://github.com/gunesmes/testhive-django-unittest-behave-se
lenium-newman-locust-jenkins
+http://www.testrisk.com/2015/09/performance-testing-on-ci-locus
t-is.html
+http://www.testrisk.com/2016/01/performance-testing-on-ci-integr
ation.html


Connect Me on
LinkedIn


Download
Presentation