Monitor-Based Testing of Elastic Cloud Computing Applications

A test procedure that monitors the "elasticity states", and executes the test accordingly.

1. © AtlanMod ([email protected])© AtlanMod ([email protected])
Monitor-Based Testing of
Elastic Cloud Computing
Applications
Michel Albonico
PhD Student - AtlanMod - EMN (Nantes, France)
([email protected])
Jean-Marie Mottu
Gerson Sunyé

1
5th Int. Workshop on Large Scale Testing
Delft, Netherlands - 2016

2. © AtlanMod ([email protected])
●Cloud Computing Elasticity
●Motivation
●Test Procedure
●Experiments
●Conclusion and Future Work
Outline
2

3. © AtlanMod ([email protected])© AtlanMod ([email protected])
Cloud Computing Elasticity
3
●Cloud computing elasticity:

The ability of a cloud infrastructure/system modifying its resource
configuration according to demand.

4. © AtlanMod ([email protected])© AtlanMod ([email protected])
Cloud Computing Elasticity
4
●Thresholds:
○Scale-out threshold: maximum resource usage, e.g., 80% of CPU
usage;
○Scale-in threshold: minimum resource usage, e.g., 20% of CPU
usage;
○Used to decide when varying a resource.




Resource Allocation
Resource Demand
Scale-out Threshold
Scale-in Threshold
Scale-out Threshold Breaching
Scale-in Threshold Breaching
Time (s)
Resource
(Processors)
1
Legend
80% 0.8
20% 0.2

5. © AtlanMod ([email protected])© AtlanMod ([email protected])
Cloud Computing Elasticity
5
●Resource demand varies according to workload variations.
○Example:
■number of users increases from 1 to 2, the resource
demand doubles.




Resource Allocation
Resource Demand
Scale-out Threshold
Scale-in Threshold
Scale-out Threshold Breaching
Scale-in Threshold Breaching
Time (s)
Resource
(Processors)
1
Legend
80% 0.8
20% 0.2

6. © AtlanMod ([email protected])© AtlanMod ([email protected])
Cloud Computing Elasticity
6
●Resource demand varies over time;
●Scale-out threshold breaching.



Resource Allocation
Resource Demand
Scale-out Threshold
Scale-in Threshold
Scale-out Threshold Breaching
Scale-in Threshold Breaching
Time (s)
Resource
(Processors)
1
Legend
scale-out threshold breaching
80% 0.8
20% 0.2

7. © AtlanMod ([email protected])© AtlanMod ([email protected])
Cloud Computing Elasticity
7
●Resource demand varies over time;
●Scale-out threshold breaching;
●Scale-out reaction time;




Resource Allocation
Resource Demand
Scale-out Threshold
Scale-in Threshold
Scale-out Threshold Breaching
Scale-in Threshold Breaching
Time (s)
Resource
(Processors)
1
Legend
scale-out reaction time
80% 0.8
20% 0.2

8. © AtlanMod ([email protected])© AtlanMod ([email protected])
Cloud Computing Elasticity
8
●Resource demand varies over time;
●Scale-out threshold breaching;
●Scale-out reaction time;
●Scale-out time, then the thresholds are updated.




Resource Allocation
Resource Demand
Scale-out Threshold
Scale-in Threshold
Scale-out Threshold Breaching
Scale-in Threshold Breaching
Time (s)
Resource
(Processors)
2
1
80% 0.8
Legend
scale-out time
80% 1.6
20% 0.2
20% 0.4

9. © AtlanMod ([email protected])© AtlanMod ([email protected])
Cloud Computing Elasticity
9
●Scale-in:
○Scale-in threshold breaching;
○Scale-in reaction time (resource is no longer available);
■Thresholds reconfiguration.
○Scale-in time.




Resource Allocation
Resource Demand
Scale-out Threshold
Scale-in Threshold
Scale-out Threshold Breaching
Scale-in Threshold Breaching
Time (s)
Resource
(Processors)
2
1
Legend
scale-in time
scale-in reaction time
80% 0.8
80% 1.6
20% 0.2
20% 0.4

10. © AtlanMod ([email protected])© AtlanMod ([email protected])
Cloud Computing Elasticity
10
●Elasticity states transition.




scale-out
threshold
breaching
scale-in
threshold
breaching

11. © AtlanMod ([email protected])© AtlanMod ([email protected])
●Elasticity states transition.







●Related work only test during the ready state;
●Scaling states:
○Considerable time: in our experiments, scaling-out takes more
than 90 seconds (Amazon EC2);
○Great part of the adaptation tasks: replication data, leader
election, etc.




Motivation
11

12. © AtlanMod ([email protected])© AtlanMod ([email protected])
Test Procedure
●Test cloud systems during all the elasticity states;
●Execute tests dynamically:
○Associate test cases to a set of elasticity states;
○Execute the test according to the current elasticity state.

●Test execution:
○Periodically monitor the resource during the test execution;
■Current elasticity state.
○(Re)-execute the associated test cases during the current
elasticity state.




12

13. © AtlanMod ([email protected])© AtlanMod ([email protected])
Experiments
●Research questions (answered by the experiments):

1.Is it necessary to run the test during different elasticity states?
a.Does a cloud system react distinctly depending on the
elasticity state?

2.Is it possible to execute the test during different elasticity states
and to assign the test verdicts accordingly?



13

14. © AtlanMod ([email protected])© AtlanMod ([email protected])
Experiments
Question 1: system behavior during different elasticity state.

●First experiment:
Measure the performance of a cloud system during different
elasticity states.

○Manually executed;

○Workload (50% read / 50% write);

○2500 operations per second (ops).





14

15. © AtlanMod ([email protected])© AtlanMod ([email protected])
Experiments
●First experiment results:
○2000 ops: covers all the performance drops;
○Elasticity states extracted from the log files;

RQ1:
It is necessary to run the test during different elasticity states.


15
Performance - Operations per Second (ops)
Minimal Performance
Measured Performance
R R R R R RSISISO SO SOSI
200 400
1000 1200
800
600
Time (s)

16. © AtlanMod ([email protected])© AtlanMod ([email protected])
Question 2: test execution during different elasticity states + test
verdicts assignment.

●Second experiment: (same workload)
○We use our test procedure;
○We monitor the elasticity states throughout the test
execution;
○Test Case:
■answered operation >= 2000 ops -> pass
■otherwise -> fail

○Same test case associated to every elasticity state.
■Test case re-executed throughout the cloud system
execution.





Experiments
16

17. © AtlanMod ([email protected])© AtlanMod ([email protected])
Experiments
●Result of the second experiment:
○Test through different elasticity states;
○Assign test verdicts to different elasticity states;
■Proportional to the previous experiment (correct elasticity
states).
RQ2:
It is possible to execute the test according to the elasticity state, and
we are able to assign the test verdicts correctly.






17

18. © AtlanMod ([email protected])© AtlanMod ([email protected])
Conclusion and Future Work
●Identify all the performance problems;
●Assign the test verdicts to the correct elasticity states (at
runtime);
●Address the scaling states, which are not addressed by related
work;
●Future work:
○Write functional test cases;
○Apply to other study cases;
○Generate test cases based on elasticity states.







18

19. © AtlanMod ([email protected])© AtlanMod ([email protected])
Monitor-Based Testing of
Elastic Cloud Computing
Applications
Michel Albonico
PhD Student - AtlanMod - EMN (Nantes, France)
([email protected])
Jean-Marie Mottu
Gerson Sunyé

19
5th Int. Workshop on Large Scale Testing
Delft, Netherlands - 2016